2023-04-03 02:53:34,726 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022 2023-04-03 02:53:34,741 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-03 02:53:34,758 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-03 02:53:34,759 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/cluster_db440580-0945-d2b2-2bc7-6fe721462712, deleteOnExit=true 2023-04-03 02:53:34,759 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-03 02:53:34,760 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/test.cache.data in system properties and HBase conf 2023-04-03 02:53:34,760 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/hadoop.tmp.dir in system properties and HBase conf 2023-04-03 02:53:34,761 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/hadoop.log.dir in system properties and HBase conf 2023-04-03 02:53:34,762 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-03 02:53:34,762 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-03 02:53:34,762 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-03 02:53:34,882 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-03 02:53:35,311 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-03 02:53:35,317 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-03 02:53:35,317 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-03 02:53:35,318 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-03 02:53:35,318 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-03 02:53:35,319 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-03 02:53:35,319 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-03 02:53:35,319 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-03 02:53:35,320 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-03 02:53:35,320 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-03 02:53:35,321 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/nfs.dump.dir in system properties and HBase conf 2023-04-03 02:53:35,321 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/java.io.tmpdir in system properties and HBase conf 2023-04-03 02:53:35,322 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-03 02:53:35,322 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-03 02:53:35,322 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-03 02:53:35,956 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-03 02:53:35,962 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-03 02:53:36,266 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-03 02:53:36,448 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-03 02:53:36,468 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-03 02:53:36,511 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-03 02:53:36,543 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/java.io.tmpdir/Jetty_localhost_34805_hdfs____pw6s4m/webapp 2023-04-03 02:53:36,689 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34805 2023-04-03 02:53:36,704 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-03 02:53:36,705 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-03 02:53:37,293 WARN [Listener at localhost/37791] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-03 02:53:37,375 WARN [Listener at localhost/37791] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-03 02:53:37,400 WARN [Listener at localhost/37791] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-03 02:53:37,408 INFO [Listener at localhost/37791] log.Slf4jLog(67): jetty-6.1.26 2023-04-03 02:53:37,418 INFO [Listener at localhost/37791] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/java.io.tmpdir/Jetty_localhost_44273_datanode____r6e039/webapp 2023-04-03 02:53:37,521 INFO [Listener at localhost/37791] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44273 2023-04-03 02:53:37,895 WARN [Listener at localhost/46853] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-03 02:53:39,081 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8913899a91906c7: Processing first storage report for DS-ba09eb5d-e032-4ef7-b0eb-8d8c3889969f from datanode 328bbd6e-f653-4143-b18d-0bb4dc77cf58 2023-04-03 02:53:39,082 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8913899a91906c7: from storage DS-ba09eb5d-e032-4ef7-b0eb-8d8c3889969f node DatanodeRegistration(127.0.0.1:39467, datanodeUuid=328bbd6e-f653-4143-b18d-0bb4dc77cf58, infoPort=33113, infoSecurePort=0, ipcPort=46853, storageInfo=lv=-57;cid=testClusterID;nsid=1938940320;c=1680490416035), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-03 02:53:39,082 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8913899a91906c7: Processing first storage report for DS-d6d503ba-4275-4d6e-91d4-71885009ced4 from datanode 328bbd6e-f653-4143-b18d-0bb4dc77cf58 2023-04-03 02:53:39,082 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8913899a91906c7: from storage DS-d6d503ba-4275-4d6e-91d4-71885009ced4 node DatanodeRegistration(127.0.0.1:39467, datanodeUuid=328bbd6e-f653-4143-b18d-0bb4dc77cf58, infoPort=33113, infoSecurePort=0, ipcPort=46853, storageInfo=lv=-57;cid=testClusterID;nsid=1938940320;c=1680490416035), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-03 02:53:39,149 DEBUG [Listener at localhost/46853] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022 2023-04-03 02:53:39,210 INFO [Listener at localhost/46853] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/cluster_db440580-0945-d2b2-2bc7-6fe721462712/zookeeper_0, clientPort=52506, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/cluster_db440580-0945-d2b2-2bc7-6fe721462712/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/cluster_db440580-0945-d2b2-2bc7-6fe721462712/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-03 02:53:39,223 INFO [Listener at localhost/46853] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=52506 2023-04-03 02:53:39,230 INFO [Listener at localhost/46853] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 02:53:39,232 INFO [Listener at localhost/46853] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 02:53:39,887 INFO [Listener at localhost/46853] util.FSUtils(471): Created version file at hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c with version=8 2023-04-03 02:53:39,888 INFO [Listener at localhost/46853] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/hbase-staging 2023-04-03 02:53:40,236 INFO [Listener at localhost/46853] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-03 02:53:40,743 INFO [Listener at localhost/46853] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-03 02:53:40,778 INFO [Listener at localhost/46853] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 02:53:40,779 INFO [Listener at localhost/46853] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-03 02:53:40,779 INFO [Listener at localhost/46853] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-03 02:53:40,779 INFO [Listener at localhost/46853] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 02:53:40,779 INFO [Listener at localhost/46853] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-03 02:53:40,939 INFO [Listener at localhost/46853] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-03 02:53:41,036 DEBUG [Listener at localhost/46853] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-03 02:53:41,129 INFO [Listener at localhost/46853] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37975 2023-04-03 02:53:41,140 INFO [Listener at localhost/46853] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 02:53:41,142 INFO [Listener at localhost/46853] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 02:53:41,166 INFO [Listener at localhost/46853] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37975 connecting to ZooKeeper ensemble=127.0.0.1:52506 2023-04-03 02:53:41,208 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:379750x0, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-03 02:53:41,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:37975-0x10147f0bdc10000 connected 2023-04-03 02:53:41,245 DEBUG [Listener at localhost/46853] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-03 02:53:41,246 DEBUG [Listener at localhost/46853] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 02:53:41,249 DEBUG [Listener at localhost/46853] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-03 02:53:41,258 DEBUG [Listener at localhost/46853] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37975 2023-04-03 02:53:41,258 DEBUG [Listener at localhost/46853] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37975 2023-04-03 02:53:41,258 DEBUG [Listener at localhost/46853] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37975 2023-04-03 02:53:41,262 DEBUG [Listener at localhost/46853] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37975 2023-04-03 02:53:41,263 DEBUG [Listener at localhost/46853] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37975 2023-04-03 02:53:41,270 INFO [Listener at localhost/46853] master.HMaster(444): hbase.rootdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c, hbase.cluster.distributed=false 2023-04-03 02:53:41,349 INFO [Listener at localhost/46853] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-03 02:53:41,350 INFO [Listener at localhost/46853] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 02:53:41,350 INFO [Listener at localhost/46853] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-03 02:53:41,350 INFO [Listener at localhost/46853] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-03 02:53:41,350 INFO [Listener at localhost/46853] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 02:53:41,350 INFO [Listener at localhost/46853] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-03 02:53:41,355 INFO [Listener at localhost/46853] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-03 02:53:41,358 INFO [Listener at localhost/46853] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43815 2023-04-03 02:53:41,361 INFO [Listener at localhost/46853] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-03 02:53:41,369 DEBUG [Listener at localhost/46853] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-03 02:53:41,371 INFO [Listener at localhost/46853] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 02:53:41,375 INFO [Listener at localhost/46853] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 02:53:41,378 INFO [Listener at localhost/46853] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43815 connecting to ZooKeeper ensemble=127.0.0.1:52506 2023-04-03 02:53:41,381 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:438150x0, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-03 02:53:41,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:43815-0x10147f0bdc10001 connected 2023-04-03 02:53:41,383 DEBUG [Listener at localhost/46853] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-03 02:53:41,384 DEBUG [Listener at localhost/46853] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 02:53:41,385 DEBUG [Listener at localhost/46853] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-03 02:53:41,388 DEBUG [Listener at localhost/46853] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43815 2023-04-03 02:53:41,388 DEBUG [Listener at localhost/46853] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43815 2023-04-03 02:53:41,390 DEBUG [Listener at localhost/46853] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43815 2023-04-03 02:53:41,390 DEBUG [Listener at localhost/46853] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43815 2023-04-03 02:53:41,391 DEBUG [Listener at localhost/46853] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43815 2023-04-03 02:53:41,396 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,37975,1680490420046 2023-04-03 02:53:41,408 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-03 02:53:41,409 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,37975,1680490420046 2023-04-03 02:53:41,434 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-03 02:53:41,435 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-03 02:53:41,435 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 02:53:41,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-03 02:53:41,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-03 02:53:41,438 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,37975,1680490420046 from backup master directory 2023-04-03 02:53:41,442 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,37975,1680490420046 2023-04-03 02:53:41,443 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-03 02:53:41,443 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-03 02:53:41,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,37975,1680490420046 2023-04-03 02:53:41,447 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-03 02:53:41,449 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-03 02:53:41,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/hbase.id with ID: 2b1e6e0a-fb00-4406-92a2-2bdf59479677 2023-04-03 02:53:41,579 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 02:53:41,595 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 02:53:41,639 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0f2e809b to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:41,679 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@463c20e8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:41,711 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 02:53:41,714 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-03 02:53:41,734 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-03 02:53:41,734 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-03 02:53:41,736 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:53:41,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:53:41,742 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-03 02:53:41,784 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/data/master/store-tmp 2023-04-03 02:53:41,820 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:53:41,820 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-03 02:53:41,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 02:53:41,821 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 02:53:41,821 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-03 02:53:41,821 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 02:53:41,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 02:53:41,821 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-03 02:53:41,823 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/WALs/jenkins-hbase4.apache.org,37975,1680490420046 2023-04-03 02:53:41,845 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37975%2C1680490420046, suffix=, logDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/WALs/jenkins-hbase4.apache.org,37975,1680490420046, archiveDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/oldWALs, maxLogs=10 2023-04-03 02:53:41,895 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39467,DS-ba09eb5d-e032-4ef7-b0eb-8d8c3889969f,DISK] 2023-04-03 02:53:41,903 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:53:41,969 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/WALs/jenkins-hbase4.apache.org,37975,1680490420046/jenkins-hbase4.apache.org%2C37975%2C1680490420046.1680490421855 2023-04-03 02:53:41,970 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39467,DS-ba09eb5d-e032-4ef7-b0eb-8d8c3889969f,DISK]] 2023-04-03 02:53:41,970 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:53:41,971 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:53:41,974 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-03 02:53:41,975 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-03 02:53:42,037 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-03 02:53:42,045 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-03 02:53:42,069 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-03 02:53:42,081 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:53:42,086 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-03 02:53:42,088 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-03 02:53:42,102 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-03 02:53:42,106 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 02:53:42,107 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74382615, jitterRate=0.10838733613491058}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-03 02:53:42,107 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-03 02:53:42,112 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-03 02:53:42,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-03 02:53:42,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-03 02:53:42,133 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-03 02:53:42,135 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-03 02:53:42,167 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-04-03 02:53:42,167 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-03 02:53:42,192 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-03 02:53:42,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-03 02:53:42,222 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-03 02:53:42,225 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-03 02:53:42,227 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-03 02:53:42,233 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-03 02:53:42,238 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-03 02:53:42,241 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 02:53:42,242 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-03 02:53:42,243 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-03 02:53:42,257 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-03 02:53:42,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-03 02:53:42,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-03 02:53:42,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 02:53:42,264 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,37975,1680490420046, sessionid=0x10147f0bdc10000, setting cluster-up flag (Was=false) 2023-04-03 02:53:42,277 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 02:53:42,283 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-03 02:53:42,285 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37975,1680490420046 2023-04-03 02:53:42,289 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 02:53:42,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-03 02:53:42,296 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,37975,1680490420046 2023-04-03 02:53:42,299 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.hbase-snapshot/.tmp 2023-04-03 02:53:42,395 INFO [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(952): ClusterId : 2b1e6e0a-fb00-4406-92a2-2bdf59479677 2023-04-03 02:53:42,400 DEBUG [RS:0;jenkins-hbase4:43815] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-03 02:53:42,405 DEBUG [RS:0;jenkins-hbase4:43815] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-03 02:53:42,405 DEBUG [RS:0;jenkins-hbase4:43815] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-03 02:53:42,409 DEBUG [RS:0;jenkins-hbase4:43815] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-03 02:53:42,410 DEBUG [RS:0;jenkins-hbase4:43815] zookeeper.ReadOnlyZKClient(139): Connect 0x7129712f to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:42,416 DEBUG [RS:0;jenkins-hbase4:43815] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fee817a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:42,416 DEBUG [RS:0;jenkins-hbase4:43815] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bd11a5f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-03 02:53:42,417 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-03 02:53:42,427 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-03 02:53:42,427 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-03 02:53:42,428 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-03 02:53:42,428 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-03 02:53:42,428 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-03 02:53:42,428 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 02:53:42,428 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-03 02:53:42,428 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 02:53:42,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680490452431 2023-04-03 02:53:42,433 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-03 02:53:42,437 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-03 02:53:42,437 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-03 02:53:42,440 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:43815 2023-04-03 02:53:42,443 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-03 02:53:42,443 INFO [RS:0;jenkins-hbase4:43815] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-03 02:53:42,443 INFO [RS:0;jenkins-hbase4:43815] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-03 02:53:42,443 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1023): About to register with Master. 2023-04-03 02:53:42,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-03 02:53:42,446 INFO [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,37975,1680490420046 with isa=jenkins-hbase4.apache.org/172.31.14.131:43815, startcode=1680490421348 2023-04-03 02:53:42,451 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-03 02:53:42,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-03 02:53:42,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-03 02:53:42,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-03 02:53:42,453 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:42,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-03 02:53:42,455 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-03 02:53:42,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-03 02:53:42,458 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-03 02:53:42,458 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-03 02:53:42,460 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680490422460,5,FailOnTimeoutGroup] 2023-04-03 02:53:42,460 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680490422460,5,FailOnTimeoutGroup] 2023-04-03 02:53:42,460 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:42,460 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-03 02:53:42,462 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:42,462 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:42,464 DEBUG [RS:0;jenkins-hbase4:43815] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-03 02:53:42,512 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42757, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-03 02:53:42,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:42,536 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c 2023-04-03 02:53:42,536 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37791 2023-04-03 02:53:42,536 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-03 02:53:42,541 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-03 02:53:42,542 DEBUG [RS:0;jenkins-hbase4:43815] zookeeper.ZKUtil(162): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:42,542 WARN [RS:0;jenkins-hbase4:43815] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-03 02:53:42,543 INFO [RS:0;jenkins-hbase4:43815] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-03 02:53:42,543 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/WALs/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:42,543 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43815,1680490421348] 2023-04-03 02:53:42,552 DEBUG [RS:0;jenkins-hbase4:43815] zookeeper.ZKUtil(162): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:42,562 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-03 02:53:42,570 INFO [RS:0;jenkins-hbase4:43815] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-03 02:53:42,590 INFO [RS:0;jenkins-hbase4:43815] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-03 02:53:42,594 INFO [RS:0;jenkins-hbase4:43815] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-03 02:53:42,595 INFO [RS:0;jenkins-hbase4:43815] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:42,596 INFO [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-03 02:53:42,604 INFO [RS:0;jenkins-hbase4:43815] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:42,604 DEBUG [RS:0;jenkins-hbase4:43815] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 02:53:42,604 DEBUG [RS:0;jenkins-hbase4:43815] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 02:53:42,604 DEBUG [RS:0;jenkins-hbase4:43815] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 02:53:42,605 DEBUG [RS:0;jenkins-hbase4:43815] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 02:53:42,605 DEBUG [RS:0;jenkins-hbase4:43815] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 02:53:42,605 DEBUG [RS:0;jenkins-hbase4:43815] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-03 02:53:42,605 DEBUG [RS:0;jenkins-hbase4:43815] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 02:53:42,605 DEBUG [RS:0;jenkins-hbase4:43815] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 02:53:42,605 DEBUG [RS:0;jenkins-hbase4:43815] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 02:53:42,605 DEBUG [RS:0;jenkins-hbase4:43815] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 02:53:42,606 INFO [RS:0;jenkins-hbase4:43815] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:42,606 INFO [RS:0;jenkins-hbase4:43815] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:42,606 INFO [RS:0;jenkins-hbase4:43815] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:42,622 INFO [RS:0;jenkins-hbase4:43815] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-03 02:53:42,625 INFO [RS:0;jenkins-hbase4:43815] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43815,1680490421348-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:42,640 INFO [RS:0;jenkins-hbase4:43815] regionserver.Replication(203): jenkins-hbase4.apache.org,43815,1680490421348 started 2023-04-03 02:53:42,640 INFO [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,43815,1680490421348, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43815, sessionid=0x10147f0bdc10001 2023-04-03 02:53:42,641 DEBUG [RS:0;jenkins-hbase4:43815] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-03 02:53:42,641 DEBUG [RS:0;jenkins-hbase4:43815] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:42,641 DEBUG [RS:0;jenkins-hbase4:43815] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43815,1680490421348' 2023-04-03 02:53:42,641 DEBUG [RS:0;jenkins-hbase4:43815] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:42,641 DEBUG [RS:0;jenkins-hbase4:43815] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:42,642 DEBUG [RS:0;jenkins-hbase4:43815] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-03 02:53:42,642 DEBUG [RS:0;jenkins-hbase4:43815] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-03 02:53:42,642 DEBUG [RS:0;jenkins-hbase4:43815] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:42,642 DEBUG [RS:0;jenkins-hbase4:43815] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43815,1680490421348' 2023-04-03 02:53:42,642 DEBUG [RS:0;jenkins-hbase4:43815] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-03 02:53:42,642 DEBUG [RS:0;jenkins-hbase4:43815] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-03 02:53:42,643 DEBUG [RS:0;jenkins-hbase4:43815] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-03 02:53:42,643 INFO [RS:0;jenkins-hbase4:43815] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-03 02:53:42,643 INFO [RS:0;jenkins-hbase4:43815] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-03 02:53:42,754 INFO [RS:0;jenkins-hbase4:43815] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43815%2C1680490421348, suffix=, logDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/WALs/jenkins-hbase4.apache.org,43815,1680490421348, archiveDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/oldWALs, maxLogs=32 2023-04-03 02:53:42,772 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39467,DS-ba09eb5d-e032-4ef7-b0eb-8d8c3889969f,DISK] 2023-04-03 02:53:42,775 INFO [RS:0;jenkins-hbase4:43815] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/WALs/jenkins-hbase4.apache.org,43815,1680490421348/jenkins-hbase4.apache.org%2C43815%2C1680490421348.1680490422756 2023-04-03 02:53:42,776 DEBUG [RS:0;jenkins-hbase4:43815] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39467,DS-ba09eb5d-e032-4ef7-b0eb-8d8c3889969f,DISK]] 2023-04-03 02:53:42,877 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-03 02:53:42,878 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-03 02:53:42,879 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c 2023-04-03 02:53:42,892 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:53:42,894 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-03 02:53:42,897 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/info 2023-04-03 02:53:42,898 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-03 02:53:42,899 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:53:42,899 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-03 02:53:42,902 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/rep_barrier 2023-04-03 02:53:42,903 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-03 02:53:42,904 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:53:42,904 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-03 02:53:42,906 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/table 2023-04-03 02:53:42,907 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-03 02:53:42,908 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:53:42,910 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740 2023-04-03 02:53:42,910 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740 2023-04-03 02:53:42,914 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 02:53:42,915 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-03 02:53:42,919 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 02:53:42,920 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61413235, jitterRate=-0.08487148582935333}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 02:53:42,920 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-03 02:53:42,920 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-03 02:53:42,920 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-03 02:53:42,920 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-03 02:53:42,920 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-03 02:53:42,920 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-03 02:53:42,922 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-03 02:53:42,922 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-03 02:53:42,926 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-03 02:53:42,926 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-03 02:53:42,934 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-03 02:53:42,945 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-03 02:53:42,947 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-03 02:53:43,099 DEBUG [jenkins-hbase4:37975] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-03 02:53:43,103 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43815,1680490421348, state=OPENING 2023-04-03 02:53:43,112 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-03 02:53:43,114 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 02:53:43,115 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-03 02:53:43,118 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:53:43,304 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:43,306 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-03 02:53:43,311 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60564, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-03 02:53:43,322 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-03 02:53:43,323 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-03 02:53:43,326 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43815%2C1680490421348.meta, suffix=.meta, logDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/WALs/jenkins-hbase4.apache.org,43815,1680490421348, archiveDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/oldWALs, maxLogs=32 2023-04-03 02:53:43,343 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39467,DS-ba09eb5d-e032-4ef7-b0eb-8d8c3889969f,DISK] 2023-04-03 02:53:43,350 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/WALs/jenkins-hbase4.apache.org,43815,1680490421348/jenkins-hbase4.apache.org%2C43815%2C1680490421348.meta.1680490423328.meta 2023-04-03 02:53:43,350 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39467,DS-ba09eb5d-e032-4ef7-b0eb-8d8c3889969f,DISK]] 2023-04-03 02:53:43,350 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:53:43,353 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-03 02:53:43,373 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-03 02:53:43,378 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-03 02:53:43,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-03 02:53:43,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:53:43,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-03 02:53:43,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-03 02:53:43,389 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-03 02:53:43,390 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/info 2023-04-03 02:53:43,391 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/info 2023-04-03 02:53:43,391 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-03 02:53:43,392 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:53:43,392 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-03 02:53:43,394 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/rep_barrier 2023-04-03 02:53:43,394 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/rep_barrier 2023-04-03 02:53:43,395 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-03 02:53:43,396 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:53:43,396 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-03 02:53:43,398 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/table 2023-04-03 02:53:43,398 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/table 2023-04-03 02:53:43,399 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-03 02:53:43,400 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:53:43,402 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740 2023-04-03 02:53:43,405 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740 2023-04-03 02:53:43,409 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 02:53:43,413 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-03 02:53:43,414 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63363627, jitterRate=-0.055808380246162415}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 02:53:43,414 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-03 02:53:43,423 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680490423296 2023-04-03 02:53:43,440 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-03 02:53:43,440 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-03 02:53:43,441 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,43815,1680490421348, state=OPEN 2023-04-03 02:53:43,444 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-03 02:53:43,444 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-03 02:53:43,448 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-03 02:53:43,448 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,43815,1680490421348 in 327 msec 2023-04-03 02:53:43,452 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-03 02:53:43,452 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 516 msec 2023-04-03 02:53:43,457 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1110 sec 2023-04-03 02:53:43,457 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680490423457, completionTime=-1 2023-04-03 02:53:43,457 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-03 02:53:43,458 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-03 02:53:43,518 DEBUG [hconnection-0x7564d4f3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:43,522 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47568, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:43,540 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-03 02:53:43,540 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680490483540 2023-04-03 02:53:43,540 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680490543540 2023-04-03 02:53:43,540 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 82 msec 2023-04-03 02:53:43,561 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37975,1680490420046-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:43,562 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37975,1680490420046-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:43,562 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37975,1680490420046-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:43,563 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:37975, period=300000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:43,564 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-03 02:53:43,569 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-03 02:53:43,576 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-03 02:53:43,578 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-03 02:53:43,586 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-03 02:53:43,589 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 02:53:43,591 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 02:53:43,611 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:53:43,613 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3 empty. 2023-04-03 02:53:43,614 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:53:43,614 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-03 02:53:43,634 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-03 02:53:43,637 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 624516272efe3dcb8cad0effd67e2ce3, NAME => 'hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp 2023-04-03 02:53:44,051 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:53:44,051 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 624516272efe3dcb8cad0effd67e2ce3, disabling compactions & flushes 2023-04-03 02:53:44,051 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:53:44,051 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:53:44,051 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. after waiting 0 ms 2023-04-03 02:53:44,051 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:53:44,051 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:53:44,051 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 624516272efe3dcb8cad0effd67e2ce3: 2023-04-03 02:53:44,056 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 02:53:44,071 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680490424059"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490424059"}]},"ts":"1680490424059"} 2023-04-03 02:53:44,096 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 02:53:44,098 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 02:53:44,102 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490424098"}]},"ts":"1680490424098"} 2023-04-03 02:53:44,106 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-03 02:53:44,114 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=624516272efe3dcb8cad0effd67e2ce3, ASSIGN}] 2023-04-03 02:53:44,117 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=624516272efe3dcb8cad0effd67e2ce3, ASSIGN 2023-04-03 02:53:44,118 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=624516272efe3dcb8cad0effd67e2ce3, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43815,1680490421348; forceNewPlan=false, retain=false 2023-04-03 02:53:44,269 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=624516272efe3dcb8cad0effd67e2ce3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:44,270 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680490424269"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490424269"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490424269"}]},"ts":"1680490424269"} 2023-04-03 02:53:44,274 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 624516272efe3dcb8cad0effd67e2ce3, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:53:44,434 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:53:44,435 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 624516272efe3dcb8cad0effd67e2ce3, NAME => 'hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3.', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:53:44,436 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:53:44,436 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:53:44,436 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:53:44,436 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:53:44,438 INFO [StoreOpener-624516272efe3dcb8cad0effd67e2ce3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:53:44,441 DEBUG [StoreOpener-624516272efe3dcb8cad0effd67e2ce3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3/info 2023-04-03 02:53:44,441 DEBUG [StoreOpener-624516272efe3dcb8cad0effd67e2ce3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3/info 2023-04-03 02:53:44,441 INFO [StoreOpener-624516272efe3dcb8cad0effd67e2ce3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 624516272efe3dcb8cad0effd67e2ce3 columnFamilyName info 2023-04-03 02:53:44,442 INFO [StoreOpener-624516272efe3dcb8cad0effd67e2ce3-1] regionserver.HStore(310): Store=624516272efe3dcb8cad0effd67e2ce3/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:53:44,444 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:53:44,445 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:53:44,450 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:53:44,454 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 02:53:44,455 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 624516272efe3dcb8cad0effd67e2ce3; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66222111, jitterRate=-0.01321364939212799}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-03 02:53:44,455 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 624516272efe3dcb8cad0effd67e2ce3: 2023-04-03 02:53:44,457 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3., pid=6, masterSystemTime=1680490424428 2023-04-03 02:53:44,461 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:53:44,461 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:53:44,463 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=624516272efe3dcb8cad0effd67e2ce3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:44,463 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680490424462"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680490424462"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680490424462"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680490424462"}]},"ts":"1680490424462"} 2023-04-03 02:53:44,470 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-03 02:53:44,470 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 624516272efe3dcb8cad0effd67e2ce3, server=jenkins-hbase4.apache.org,43815,1680490421348 in 193 msec 2023-04-03 02:53:44,473 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-03 02:53:44,473 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=624516272efe3dcb8cad0effd67e2ce3, ASSIGN in 356 msec 2023-04-03 02:53:44,474 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 02:53:44,475 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490424475"}]},"ts":"1680490424475"} 2023-04-03 02:53:44,477 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-03 02:53:44,480 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 02:53:44,483 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 902 msec 2023-04-03 02:53:44,489 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-03 02:53:44,490 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-03 02:53:44,490 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 02:53:44,526 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-03 02:53:44,541 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-03 02:53:44,546 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-04-03 02:53:44,550 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-03 02:53:44,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-03 02:53:44,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 14 msec 2023-04-03 02:53:44,578 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-03 02:53:44,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-03 02:53:44,582 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.138sec 2023-04-03 02:53:44,583 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-03 02:53:44,585 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-03 02:53:44,585 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-03 02:53:44,586 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37975,1680490420046-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-03 02:53:44,586 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37975,1680490420046-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-03 02:53:44,597 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-03 02:53:44,601 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x0f3002be to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:44,606 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e324982, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:44,628 DEBUG [hconnection-0x10a3ae0a-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:44,644 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47570, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:44,655 INFO [Listener at localhost/46853] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,37975,1680490420046 2023-04-03 02:53:44,677 INFO [Listener at localhost/46853] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=442, ProcessCount=170, AvailableMemoryMB=4337 2023-04-03 02:53:44,692 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 02:53:44,695 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45086, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 02:53:44,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 02:53:44,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 02:53:44,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 02:53:44,712 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 02:53:44,714 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 02:53:44,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-03 02:53:44,718 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:44,719 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4 empty. 2023-04-03 02:53:44,720 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:44,721 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:53:44,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 02:53:44,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 02:53:45,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 02:53:45,139 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 02:53:45,141 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 459f839097be2d0597a0a35d5312fdc4, NAME => 'TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp 2023-04-03 02:53:45,152 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:53:45,152 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 459f839097be2d0597a0a35d5312fdc4, disabling compactions & flushes 2023-04-03 02:53:45,152 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:45,152 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:45,152 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. after waiting 0 ms 2023-04-03 02:53:45,152 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:45,152 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:45,153 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:45,156 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 02:53:45,158 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490425158"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490425158"}]},"ts":"1680490425158"} 2023-04-03 02:53:45,161 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 02:53:45,162 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 02:53:45,162 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490425162"}]},"ts":"1680490425162"} 2023-04-03 02:53:45,164 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 02:53:45,168 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=459f839097be2d0597a0a35d5312fdc4, ASSIGN}] 2023-04-03 02:53:45,170 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=459f839097be2d0597a0a35d5312fdc4, ASSIGN 2023-04-03 02:53:45,171 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=459f839097be2d0597a0a35d5312fdc4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43815,1680490421348; forceNewPlan=false, retain=false 2023-04-03 02:53:45,323 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=459f839097be2d0597a0a35d5312fdc4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:45,323 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490425323"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490425323"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490425323"}]},"ts":"1680490425323"} 2023-04-03 02:53:45,326 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:53:45,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 02:53:45,485 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:45,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 459f839097be2d0597a0a35d5312fdc4, NAME => 'TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:53:45,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:45,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:53:45,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:45,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:45,488 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:45,490 DEBUG [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A 2023-04-03 02:53:45,490 DEBUG [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A 2023-04-03 02:53:45,491 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:53:45,492 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 459f839097be2d0597a0a35d5312fdc4 columnFamilyName A 2023-04-03 02:53:45,493 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] regionserver.HStore(310): Store=459f839097be2d0597a0a35d5312fdc4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:53:45,493 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:45,495 DEBUG [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B 2023-04-03 02:53:45,495 DEBUG [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B 2023-04-03 02:53:45,495 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:53:45,495 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 459f839097be2d0597a0a35d5312fdc4 columnFamilyName B 2023-04-03 02:53:45,496 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] regionserver.HStore(310): Store=459f839097be2d0597a0a35d5312fdc4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:53:45,496 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:45,498 DEBUG [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C 2023-04-03 02:53:45,498 DEBUG [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C 2023-04-03 02:53:45,498 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:53:45,498 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 459f839097be2d0597a0a35d5312fdc4 columnFamilyName C 2023-04-03 02:53:45,499 INFO [StoreOpener-459f839097be2d0597a0a35d5312fdc4-1] regionserver.HStore(310): Store=459f839097be2d0597a0a35d5312fdc4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:53:45,500 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:45,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:45,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:45,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 02:53:45,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:45,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 02:53:45,511 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 459f839097be2d0597a0a35d5312fdc4; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70467530, jitterRate=0.05004802346229553}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 02:53:45,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:45,512 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., pid=11, masterSystemTime=1680490425479 2023-04-03 02:53:45,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:45,515 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:45,515 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=459f839097be2d0597a0a35d5312fdc4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:45,516 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490425515"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680490425515"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680490425515"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680490425515"}]},"ts":"1680490425515"} 2023-04-03 02:53:45,521 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-03 02:53:45,521 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 in 192 msec 2023-04-03 02:53:45,525 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-03 02:53:45,525 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=459f839097be2d0597a0a35d5312fdc4, ASSIGN in 353 msec 2023-04-03 02:53:45,526 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 02:53:45,526 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490425526"}]},"ts":"1680490425526"} 2023-04-03 02:53:45,529 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 02:53:45,531 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 02:53:45,533 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 824 msec 2023-04-03 02:53:45,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 02:53:45,840 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-03 02:53:45,845 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x08c3f619 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:45,849 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b17e1c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:45,852 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:45,856 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47574, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:45,860 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 02:53:45,862 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 02:53:45,871 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:45,877 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56c8e45e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:45,878 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x56223d42 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:45,884 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:45,884 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:45,891 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038a026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:45,891 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x2de4d1f6 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:45,895 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:45,896 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x02c57e1b to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:45,900 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dbb9ac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:45,901 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:45,905 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:45,905 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x59d6b6ea to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:45,947 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2879e2e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:45,948 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:45,952 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:45,952 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x63871a9d to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:53:45,956 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bd6fc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:53:45,966 DEBUG [hconnection-0x38232cc9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:45,966 DEBUG [hconnection-0x6de4daf9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:45,967 DEBUG [hconnection-0x23262a5d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:45,970 DEBUG [hconnection-0x2d5134f6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:45,970 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47586, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:45,971 DEBUG [hconnection-0x6e108031-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:45,971 DEBUG [hconnection-0x3c8b895e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:45,971 DEBUG [hconnection-0x1c2d29dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:45,973 DEBUG [hconnection-0x54c5b576-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:45,973 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:45,973 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47614, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:45,974 DEBUG [hconnection-0x67c52f02-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:53:45,979 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:45,979 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47654, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:45,980 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:45,988 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:45,990 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:45,991 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47670, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:53:45,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:46,006 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:46,006 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:46,006 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:46,006 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:46,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:46,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:46,009 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:46,009 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:46,009 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,009 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:46,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:46,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:46,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:46,025 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:46,026 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:46,028 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:46,028 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:46,034 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:46,034 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:46,036 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 02:53:46,055 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:46,056 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:46,056 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:46,057 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:46,057 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:46,057 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:46,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:46,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:46,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680490426110, max timestamp=1680490426135Num uniques 859; 2023-04-03 02:53:46,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680490426110, max timestamp=1680490426137Num uniques 859; 2023-04-03 02:53:46,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-03 02:53:46,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680490426110, max timestamp=1680490426140Num uniques 859; 2023-04-03 02:53:46,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-03 02:53:46,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-03 02:53:46,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680490426168, max timestamp=1680490426213Num uniques 1009; 2023-04-03 02:53:46,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-04-03 02:53:46,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490426187, max timestamp=1680490426213Num uniques 1009; 2023-04-03 02:53:46,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-03 02:53:46,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680490426187, max timestamp=1680490426219Num uniques 1009; 2023-04-03 02:53:46,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-03 02:53:46,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490486288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490486291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680490426271, max timestamp=1680490426290Num uniques 1077; 2023-04-03 02:53:46,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49312, new segment data size=6870, suffix heap size=139976,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=150 2023-04-03 02:53:46,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680490426271, max timestamp=1680490426290Num uniques 1009; 2023-04-03 02:53:46,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-03 02:53:46,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680490426271, max timestamp=1680490426290Num uniques 1027; 2023-04-03 02:53:46,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-03 02:53:46,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490486418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680490426402, max timestamp=1680490426419Num uniques 1009; 2023-04-03 02:53:46,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490486425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-03 02:53:46,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680490426402, max timestamp=1680490426419Num uniques 1009; 2023-04-03 02:53:46,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-03 02:53:46,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680490426402, max timestamp=1680490426419Num uniques 1009; 2023-04-03 02:53:46,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-03 02:53:46,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680490426455, max timestamp=1680490426479Num uniques 1009; 2023-04-03 02:53:46,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-04-03 02:53:46,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680490426455, max timestamp=1680490426479Num uniques 1009; 2023-04-03 02:53:46,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-04-03 02:53:46,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680490426455, max timestamp=1680490426479Num uniques 1009; 2023-04-03 02:53:46,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-04-03 02:53:46,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16210, totalHeapSize=37056, min timestamp=1680490426553, max timestamp=1680490426578Num uniques 1104; 2023-04-03 02:53:46,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=16210, suffix heap size=142784,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=354 2023-04-03 02:53:46,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16205, totalHeapSize=37056, min timestamp=1680490426553, max timestamp=1680490426578Num uniques 1104; 2023-04-03 02:53:46,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50555, new segment data size=16205, suffix heap size=142784,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=354 2023-04-03 02:53:46,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16210, totalHeapSize=37056, min timestamp=1680490426553, max timestamp=1680490426578Num uniques 1104; 2023-04-03 02:53:46,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=16210, suffix heap size=142784,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=354 2023-04-03 02:53:46,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490486623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490486625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490486631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490486633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490486633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,680 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/a2d1e18d90ad443f858af2775e7aecf5 2023-04-03 02:53:46,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490486737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490486738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490486740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490486740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490486742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490486945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490486945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490486946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490486947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:46,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:46,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490486948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,211 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/d42872c9839b4b008973d3deecd891fd 2023-04-03 02:53:47,248 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/477638a8854f4ae387fade412079519f 2023-04-03 02:53:47,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490487248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490487251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490487253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490487253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490487254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,262 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/a2d1e18d90ad443f858af2775e7aecf5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a2d1e18d90ad443f858af2775e7aecf5 2023-04-03 02:53:47,274 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a2d1e18d90ad443f858af2775e7aecf5, entries=100, sequenceid=9, filesize=9.3 K 2023-04-03 02:53:47,277 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/d42872c9839b4b008973d3deecd891fd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d42872c9839b4b008973d3deecd891fd 2023-04-03 02:53:47,293 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d42872c9839b4b008973d3deecd891fd, entries=100, sequenceid=9, filesize=9.3 K 2023-04-03 02:53:47,297 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/477638a8854f4ae387fade412079519f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477638a8854f4ae387fade412079519f 2023-04-03 02:53:47,310 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477638a8854f4ae387fade412079519f, entries=100, sequenceid=9, filesize=9.3 K 2023-04-03 02:53:47,313 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 459f839097be2d0597a0a35d5312fdc4 in 1276ms, sequenceid=9, compaction requested=false 2023-04-03 02:53:47,315 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-03 02:53:47,319 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:47,319 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:47,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:47,321 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:47,321 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,321 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:47,321 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:47,324 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:47,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:47,325 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:47,325 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:47,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:47,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73222e4e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:47,328 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:47,328 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:47,335 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:47,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:47,335 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:47,335 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,336 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:47,340 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:47,340 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:47,340 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:47,343 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:47,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:47,348 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:47,348 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:47,348 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:47,348 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:47,348 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:47,354 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:47,354 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:47,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:47,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:47,356 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:47,358 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:47,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,361 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:47,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:47,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,377 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:47,377 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:47,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:47,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:47,378 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,378 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:47,378 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:47,378 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,378 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:47,378 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:47,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:47,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:47,378 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:47,380 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:47,386 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:47,387 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:47,487 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:47,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:47,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:47,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:47,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:47,509 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:47,509 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:47,513 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:47,513 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:47,513 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:47,514 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:47,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:47,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:47,519 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:47,519 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,519 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:47,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:47,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:47,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:47,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:47,523 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:47,523 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:47,525 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:47,525 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:47,525 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=154.31 KB heapSize=426.47 KB 2023-04-03 02:53:47,526 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:47,526 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:47,526 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:47,526 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:47,527 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:47,527 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:47,592 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/6ac3460e5021409ba0a77c82b449ea0b 2023-04-03 02:53:47,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:47,760 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:47,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490487763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490487765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490487767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490487771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490487769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490487874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490487875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490487876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:47,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:47,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490487877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490488083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490488084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490488085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490488086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,106 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/f21837a5e6234deb9202f5f7e9168457 2023-04-03 02:53:48,172 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/3423eedc424d461f8acea47ccd2c5644 2023-04-03 02:53:48,185 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/6ac3460e5021409ba0a77c82b449ea0b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6ac3460e5021409ba0a77c82b449ea0b 2023-04-03 02:53:48,195 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6ac3460e5021409ba0a77c82b449ea0b, entries=150, sequenceid=131, filesize=11.7 K 2023-04-03 02:53:48,196 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/f21837a5e6234deb9202f5f7e9168457 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f21837a5e6234deb9202f5f7e9168457 2023-04-03 02:53:48,210 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f21837a5e6234deb9202f5f7e9168457, entries=150, sequenceid=131, filesize=11.7 K 2023-04-03 02:53:48,216 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/3423eedc424d461f8acea47ccd2c5644 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3423eedc424d461f8acea47ccd2c5644 2023-04-03 02:53:48,237 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3423eedc424d461f8acea47ccd2c5644, entries=150, sequenceid=131, filesize=11.7 K 2023-04-03 02:53:48,239 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~426.42 KB/436656, currentSize=33.54 KB/34350 for 459f839097be2d0597a0a35d5312fdc4 in 714ms, sequenceid=131, compaction requested=false 2023-04-03 02:53:48,241 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:48,241 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:48,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:48,245 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:48,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:48,248 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,248 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:48,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:48,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c052dc[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:48,251 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:48,251 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,253 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,253 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:48,253 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:48,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:48,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:48,258 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:48,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:48,265 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:48,265 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:48,265 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:48,265 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:48,265 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:48,274 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,274 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:48,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:48,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,276 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,281 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,294 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,294 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,294 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,294 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,294 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,294 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:48,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:48,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:48,295 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:48,295 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:48,294 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,294 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:48,296 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:48,295 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:48,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:48,395 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:48,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:48,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:48,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:48,422 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:53:48,426 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:48,426 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:48,426 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:48,426 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:48,426 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:48,426 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:48,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:48,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:48,431 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:48,431 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:48,440 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:48,441 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:48,443 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,443 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,445 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,445 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:48,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:48,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:48,450 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,450 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:48,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:48,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490488466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490488471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,478 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:48,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490488471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490488478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:48,489 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:48,489 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:48,489 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:48,489 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:48,492 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:48,492 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:48,493 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,493 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:48,493 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:48,498 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:48,500 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:48,501 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/24425fbcc6de49888941d0ce35ed8af6 2023-04-03 02:53:48,501 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:48,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:48,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@608e900[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:48,509 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:48,509 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,512 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,512 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:48,512 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:48,512 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:48,517 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:48,517 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:48,517 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:48,517 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,517 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:48,517 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:48,518 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:48,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:48,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:48,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:48,518 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:48,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:48,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:48,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:48,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:48,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:48,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:48,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:48,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:48,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:48,558 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:48,558 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:48,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:48,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:48,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490488573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490488581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490488583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:48,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490488583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/81fe91c87c3e46e298d563528d1f9418 2023-04-03 02:53:48,658 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:48,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:48,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:48,682 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/1b762b3d8ea4437db9bb08effbc0e808 2023-04-03 02:53:48,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:48,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:48,684 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:48,684 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:48,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:48,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:48,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,686 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,687 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:48,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,688 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,688 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:48,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:48,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:48,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:48,690 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:48,690 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:48,693 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:48,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/24425fbcc6de49888941d0ce35ed8af6 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/24425fbcc6de49888941d0ce35ed8af6 2023-04-03 02:53:48,704 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:48,704 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:48,709 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:48,709 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:48,709 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:48,709 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:48,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:48,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:48,723 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,723 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:48,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:48,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27d67fb1[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:48,727 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:48,727 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,728 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/24425fbcc6de49888941d0ce35ed8af6, entries=150, sequenceid=144, filesize=11.8 K 2023-04-03 02:53:48,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,729 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:48,729 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:48,729 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:48,731 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,731 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:48,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/81fe91c87c3e46e298d563528d1f9418 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/81fe91c87c3e46e298d563528d1f9418 2023-04-03 02:53:48,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/81fe91c87c3e46e298d563528d1f9418, entries=150, sequenceid=144, filesize=11.8 K 2023-04-03 02:53:48,745 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,745 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:48,745 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:48,745 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:48,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/1b762b3d8ea4437db9bb08effbc0e808 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/1b762b3d8ea4437db9bb08effbc0e808 2023-04-03 02:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:48,760 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:48,760 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:48,760 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:48,760 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:48,760 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:48,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:48,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:48,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:48,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,764 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,766 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,775 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:48,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:48,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:48,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:48,776 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:48,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:48,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:48,780 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:48,780 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:48,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:48,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/1b762b3d8ea4437db9bb08effbc0e808, entries=150, sequenceid=144, filesize=11.8 K 2023-04-03 02:53:48,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 459f839097be2d0597a0a35d5312fdc4 in 361ms, sequenceid=144, compaction requested=true 2023-04-03 02:53:48,784 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:48,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:48,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:48,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:48,791 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:53:48,791 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:53:48,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:48,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490428463, max timestamp=1680490428791Num uniques 859; 2023-04-03 02:53:48,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490428463, max timestamp=1680490428791Num uniques 859; 2023-04-03 02:53:48,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:53:48,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:53:48,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490428463, max timestamp=1680490428790Num uniques 859; 2023-04-03 02:53:48,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:53:48,798 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33564 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:53:48,800 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/B is initiating minor compaction (all files) 2023-04-03 02:53:48,801 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/B in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:48,801 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d42872c9839b4b008973d3deecd891fd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f21837a5e6234deb9202f5f7e9168457, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/81fe91c87c3e46e298d563528d1f9418] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=32.8 K 2023-04-03 02:53:48,801 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=33.54 KB heapSize=86.16 KB 2023-04-03 02:53:48,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:48,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:48,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:48,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:48,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:48,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:48,803 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d42872c9839b4b008973d3deecd891fd, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680490426020 2023-04-03 02:53:48,804 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33564 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:53:48,805 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/A is initiating minor compaction (all files) 2023-04-03 02:53:48,805 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/A in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:48,805 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a2d1e18d90ad443f858af2775e7aecf5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6ac3460e5021409ba0a77c82b449ea0b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/24425fbcc6de49888941d0ce35ed8af6] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=32.8 K 2023-04-03 02:53:48,808 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f21837a5e6234deb9202f5f7e9168457, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680490426625 2023-04-03 02:53:48,808 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 81fe91c87c3e46e298d563528d1f9418, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680490428414 2023-04-03 02:53:48,813 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a2d1e18d90ad443f858af2775e7aecf5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680490426020 2023-04-03 02:53:48,815 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6ac3460e5021409ba0a77c82b449ea0b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680490426625 2023-04-03 02:53:48,816 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 24425fbcc6de49888941d0ce35ed8af6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680490428414 2023-04-03 02:53:48,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/f36d710c700348c1a412ae8fcb2ae47c 2023-04-03 02:53:48,859 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#B#compaction#10 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:48,859 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#A#compaction#11 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:48,881 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:48,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490428832, max timestamp=1680490428875Num uniques 859; 2023-04-03 02:53:48,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:53:48,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490428832, max timestamp=1680490428875Num uniques 859; 2023-04-03 02:53:48,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490428832, max timestamp=1680490428875Num uniques 859; 2023-04-03 02:53:48,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:53:48,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:48,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:53:48,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:48,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:48,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:48,904 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:48,904 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:48,916 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:48,916 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:48,917 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,917 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,926 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,926 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:48,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:48,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:48,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:48,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:48,931 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:48,939 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/d7e7a5ac701b4693954707d7ecdd1d7a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/d7e7a5ac701b4693954707d7ecdd1d7a 2023-04-03 02:53:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:48,946 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:48,947 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:48,947 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:48,947 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:48,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490428927, max timestamp=1680490428948Num uniques 1118; 2023-04-03 02:53:48,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-03 02:53:48,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490428927, max timestamp=1680490428948Num uniques 1118; 2023-04-03 02:53:48,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490428927, max timestamp=1680490428948Num uniques 1118; 2023-04-03 02:53:48,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-03 02:53:48,953 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,954 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,956 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,956 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-03 02:53:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:48,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:48,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e4755b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:48,958 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:48,958 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,959 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:48,959 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:48,959 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:48,959 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,959 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:48,959 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:48,963 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:48,963 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:48,963 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,963 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,968 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/A of 459f839097be2d0597a0a35d5312fdc4 into d7e7a5ac701b4693954707d7ecdd1d7a(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:48,968 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:48,968 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/A, priority=13, startTime=1680490428788; duration=0sec 2023-04-03 02:53:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:48,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:48,969 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:48,969 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:48,969 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:48,969 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:48,969 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:48,969 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:48,969 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:53:48,972 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,972 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,973 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:48,973 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33564 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:53:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:48,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:48,973 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/C is initiating minor compaction (all files) 2023-04-03 02:53:48,973 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/C in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:48,974 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477638a8854f4ae387fade412079519f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3423eedc424d461f8acea47ccd2c5644, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/1b762b3d8ea4437db9bb08effbc0e808] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=32.8 K 2023-04-03 02:53:48,974 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 477638a8854f4ae387fade412079519f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680490426020 2023-04-03 02:53:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:48,979 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 3423eedc424d461f8acea47ccd2c5644, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680490426625 2023-04-03 02:53:48,978 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,986 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 1b762b3d8ea4437db9bb08effbc0e808, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680490428414 2023-04-03 02:53:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,988 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,993 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 02:53:48,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:48,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:48,999 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:48,999 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:48,999 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,999 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:48,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:48,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:48,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:49,000 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:48,999 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:49,000 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:48,999 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,000 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,000 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,000 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,017 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#C#compaction#13 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:49,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680490428975, max timestamp=1680490429016Num uniques 1127; 2023-04-03 02:53:49,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-03 02:53:49,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680490428975, max timestamp=1680490429016Num uniques 1127; 2023-04-03 02:53:49,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-03 02:53:49,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680490428975, max timestamp=1680490429016Num uniques 1127; 2023-04-03 02:53:49,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-03 02:53:49,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680490429063, max timestamp=1680490429082Num uniques 1136; 2023-04-03 02:53:49,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680490429063, max timestamp=1680490429082Num uniques 1136; 2023-04-03 02:53:49,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=17676, suffix heap size=146112,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=386 2023-04-03 02:53:49,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=17676, suffix heap size=146112,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=386 2023-04-03 02:53:49,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680490429063, max timestamp=1680490429082Num uniques 1136; 2023-04-03 02:53:49,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=17676, suffix heap size=146112,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=386 2023-04-03 02:53:49,100 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:49,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:49,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:49,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:49,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:49,116 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:49,117 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:49,118 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:49,118 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:49,119 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,120 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,122 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:49,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:49,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:49,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:49,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:49,133 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:49,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:49,134 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:49,134 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:49,134 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:49,134 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:49,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:49,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:49,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,141 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34439989[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:49,144 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:49,144 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490489135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,145 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,146 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:49,146 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:49,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,146 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:49,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490489135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,146 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,146 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:49,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,149 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:49,150 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:49,150 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:49,153 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490489137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,155 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-03 02:53:49,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490489138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,156 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-03 02:53:49,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490489145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:49,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:49,159 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:49,160 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:49,160 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:49,160 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:49,160 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:49,163 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,163 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:49,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,163 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:49,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:49,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,173 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:49,173 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:49,173 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,173 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:49,173 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:49,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:49,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:49,173 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,174 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,176 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:49,176 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:49,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:49,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:49,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490489247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490489251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490489257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490489258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490489259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,276 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:49,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:49,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:49,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:49,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:49,300 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:49,300 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:49,301 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:49,301 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:49,302 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,302 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,303 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,303 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:49,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:49,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:49,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:49,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:49,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:49,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:49,313 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:49,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:49,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:49,314 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:49,314 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:49,314 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:49,314 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:49,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:49,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:49,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:49,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:49,316 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:49,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:49,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@754226e8[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:49,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:49,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:49,325 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,325 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:49,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:49,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:49,333 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:49,333 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:49,333 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:49,333 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:49,333 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:49,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/05f40bd3d7d841389a021c8e7cd5c208 2023-04-03 02:53:49,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:49,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,338 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:49,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:49,342 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,344 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,349 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:49,349 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:49,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:49,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:49,350 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,350 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,350 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,350 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:49,350 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,350 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,350 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:49,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:49,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:49,350 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:49,351 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:49,351 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:49,376 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/8d582d34951c4f65a94f43a0516ec65d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/8d582d34951c4f65a94f43a0516ec65d 2023-04-03 02:53:49,394 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/96251e9e770643eab525d12986b439cf 2023-04-03 02:53:49,394 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/B of 459f839097be2d0597a0a35d5312fdc4 into 8d582d34951c4f65a94f43a0516ec65d(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:49,395 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:49,395 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/B, priority=13, startTime=1680490428790; duration=0sec 2023-04-03 02:53:49,395 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:49,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/f36d710c700348c1a412ae8fcb2ae47c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f36d710c700348c1a412ae8fcb2ae47c 2023-04-03 02:53:49,434 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f36d710c700348c1a412ae8fcb2ae47c, entries=150, sequenceid=166, filesize=11.8 K 2023-04-03 02:53:49,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/05f40bd3d7d841389a021c8e7cd5c208 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/05f40bd3d7d841389a021c8e7cd5c208 2023-04-03 02:53:49,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/05f40bd3d7d841389a021c8e7cd5c208, entries=150, sequenceid=166, filesize=11.8 K 2023-04-03 02:53:49,453 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:49,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/96251e9e770643eab525d12986b439cf as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/96251e9e770643eab525d12986b439cf 2023-04-03 02:53:49,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:49,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:49,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490489455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490489455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:49,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:49,474 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:49,474 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:49,476 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:49,476 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:49,477 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,477 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490489469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,478 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490489470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,479 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:49,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,479 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490489470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:49,479 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:49,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:49,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/96251e9e770643eab525d12986b439cf, entries=150, sequenceid=166, filesize=11.8 K 2023-04-03 02:53:49,488 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=161.02 KB/164880 for 459f839097be2d0597a0a35d5312fdc4 in 687ms, sequenceid=166, compaction requested=false 2023-04-03 02:53:49,488 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:49,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:49,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:49,501 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:49,501 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:49,502 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:49,508 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/ea9bb2adc9d8464899dd80b7e0d47ef0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/ea9bb2adc9d8464899dd80b7e0d47ef0 2023-04-03 02:53:49,517 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:49,517 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:49,521 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:49,522 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:49,523 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=161.02 KB heapSize=443.39 KB 2023-04-03 02:53:49,523 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:49,523 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:49,523 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:49,523 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:49,523 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:49,523 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:49,528 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/C of 459f839097be2d0597a0a35d5312fdc4 into ea9bb2adc9d8464899dd80b7e0d47ef0(size=11.9 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:49,530 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:49,530 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/C, priority=13, startTime=1680490428790; duration=0sec 2023-04-03 02:53:49,530 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:49,570 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/51508d851ed749c9b78f46b9fcdc93a5 2023-04-03 02:53:49,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:49,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:49,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490489781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490489782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490489783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490489784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490489786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490489888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490489889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490489893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:49,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:49,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490489893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,020 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/6f91eb59e52c49e29a9e9eb6747a3e71 2023-04-03 02:53:50,078 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/3884fcd1c157430d9ba21117436d657d 2023-04-03 02:53:50,093 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/51508d851ed749c9b78f46b9fcdc93a5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/51508d851ed749c9b78f46b9fcdc93a5 2023-04-03 02:53:50,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490490093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490490093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490490098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490490096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,112 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/51508d851ed749c9b78f46b9fcdc93a5, entries=150, sequenceid=256, filesize=11.8 K 2023-04-03 02:53:50,116 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/6f91eb59e52c49e29a9e9eb6747a3e71 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/6f91eb59e52c49e29a9e9eb6747a3e71 2023-04-03 02:53:50,153 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/6f91eb59e52c49e29a9e9eb6747a3e71, entries=150, sequenceid=256, filesize=11.8 K 2023-04-03 02:53:50,164 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/3884fcd1c157430d9ba21117436d657d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3884fcd1c157430d9ba21117436d657d 2023-04-03 02:53:50,178 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3884fcd1c157430d9ba21117436d657d, entries=150, sequenceid=256, filesize=11.8 K 2023-04-03 02:53:50,181 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~443.34 KB/453984, currentSize=26.84 KB/27480 for 459f839097be2d0597a0a35d5312fdc4 in 658ms, sequenceid=256, compaction requested=true 2023-04-03 02:53:50,181 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:50,181 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:50,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:50,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:50,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:50,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:50,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,184 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61fde406[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:50,187 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:50,187 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,190 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:50,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:50,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:50,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:50,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:50,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:50,198 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:50,198 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:50,198 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:50,198 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:50,198 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:50,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:50,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,202 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,204 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,214 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:50,214 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:50,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:50,215 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,215 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,215 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:50,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:50,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:50,215 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,217 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,217 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,217 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,217 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,217 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:50,217 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:50,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:50,231 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-03 02:53:50,232 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-03 02:53:50,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:53:50,301 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:50,301 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:50,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:50,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:50,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:50,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:50,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:50,317 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:50,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:50,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:50,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:50,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:50,334 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:50,335 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:50,338 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:50,338 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:50,340 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,340 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,343 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,343 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:50,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:50,345 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,345 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,345 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/4355af2523024544a0658c51ca0c57e1 2023-04-03 02:53:50,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:50,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:50,352 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:50,352 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:50,357 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:50,358 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:50,358 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:50,358 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:50,358 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:50,359 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:50,359 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:50,359 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:50,359 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:50,359 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,359 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:50,359 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:50,361 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,361 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,362 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,362 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:50,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d50bcf[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:50,366 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:50,366 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,374 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:50,378 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,379 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:50,380 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,392 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,405 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/b4e43ad1591540baa83826b8d61b5754 2023-04-03 02:53:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:50,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:50,407 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:50,407 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:50,407 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:50,407 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:50,407 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:50,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,411 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,414 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,418 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:50,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:50,419 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,420 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,420 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,420 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,420 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,420 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,420 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:50,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:50,419 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:50,419 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:50,421 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:50,422 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:50,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:50,429 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/268736ad52b54e8f8f17019c4a3d2ba0 2023-04-03 02:53:50,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/4355af2523024544a0658c51ca0c57e1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/4355af2523024544a0658c51ca0c57e1 2023-04-03 02:53:50,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/4355af2523024544a0658c51ca0c57e1, entries=100, sequenceid=267, filesize=9.5 K 2023-04-03 02:53:50,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/b4e43ad1591540baa83826b8d61b5754 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/b4e43ad1591540baa83826b8d61b5754 2023-04-03 02:53:50,456 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/b4e43ad1591540baa83826b8d61b5754, entries=100, sequenceid=267, filesize=9.5 K 2023-04-03 02:53:50,457 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/268736ad52b54e8f8f17019c4a3d2ba0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/268736ad52b54e8f8f17019c4a3d2ba0 2023-04-03 02:53:50,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/268736ad52b54e8f8f17019c4a3d2ba0, entries=100, sequenceid=267, filesize=9.5 K 2023-04-03 02:53:50,478 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=60.38 KB/61830 for 459f839097be2d0597a0a35d5312fdc4 in 177ms, sequenceid=267, compaction requested=true 2023-04-03 02:53:50,478 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:50,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:50,479 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:53:50,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:50,479 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:53:50,480 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:50,480 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:50,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 02:53:50,489 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45953 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:53:50,489 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/A is initiating minor compaction (all files) 2023-04-03 02:53:50,489 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/A in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:50,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:50,491 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:50,491 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:50,491 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:50,490 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/d7e7a5ac701b4693954707d7ecdd1d7a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f36d710c700348c1a412ae8fcb2ae47c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/51508d851ed749c9b78f46b9fcdc93a5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/4355af2523024544a0658c51ca0c57e1] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=44.9 K 2023-04-03 02:53:50,491 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:50,491 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45953 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:53:50,493 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/B is initiating minor compaction (all files) 2023-04-03 02:53:50,493 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/B in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:50,493 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/8d582d34951c4f65a94f43a0516ec65d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/05f40bd3d7d841389a021c8e7cd5c208, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/6f91eb59e52c49e29a9e9eb6747a3e71, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/b4e43ad1591540baa83826b8d61b5754] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=44.9 K 2023-04-03 02:53:50,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:50,493 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting d7e7a5ac701b4693954707d7ecdd1d7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680490428414 2023-04-03 02:53:50,495 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 8d582d34951c4f65a94f43a0516ec65d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680490428414 2023-04-03 02:53:50,495 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting f36d710c700348c1a412ae8fcb2ae47c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680490428463 2023-04-03 02:53:50,506 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 05f40bd3d7d841389a021c8e7cd5c208, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680490428463 2023-04-03 02:53:50,506 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 51508d851ed749c9b78f46b9fcdc93a5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680490429134 2023-04-03 02:53:50,508 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 4355af2523024544a0658c51ca0c57e1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680490430295 2023-04-03 02:53:50,508 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 6f91eb59e52c49e29a9e9eb6747a3e71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680490429134 2023-04-03 02:53:50,509 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b4e43ad1591540baa83826b8d61b5754, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680490430295 2023-04-03 02:53:50,522 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:50,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:50,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:50,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:50,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:50,537 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:50,537 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:50,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:50,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:50,539 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,539 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,544 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:50,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:50,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:50,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:50,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:50,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:50,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:50,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:50,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:50,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:50,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:50,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:50,548 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:50,548 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:50,549 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#A#compaction#22 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:50,549 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:50,549 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:50,551 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,553 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/61211523fd62432986ff5ef2c7e3e124 2023-04-03 02:53:50,556 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#B#compaction#23 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:50,557 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,557 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,557 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:50,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff2556d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:50,560 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:50,560 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,563 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,563 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:50,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,563 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:50,566 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:50,566 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:50,566 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:50,566 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:50,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:50,570 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:50,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:50,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:50,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:50,570 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:50,573 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:50,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,576 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490490572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,579 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,582 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:50,582 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:50,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:50,582 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:50,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:50,582 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:50,582 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,583 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:50,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490490576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,583 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:50,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:50,583 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,583 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,583 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,583 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,583 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490490576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490490576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490490578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/7db6361401e14aca854714822011a9a7 2023-04-03 02:53:50,684 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:50,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:50,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490490687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490490687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:50,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:50,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:50,697 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:50,697 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:50,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490490693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:50,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:50,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490490696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,701 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,701 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490490697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,702 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:50,703 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:50,703 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,703 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:50,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:50,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:50,705 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:50,705 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:50,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:50,712 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:50,712 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:50,712 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:50,713 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:50,713 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:50,713 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:50,722 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,722 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,722 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/477f50be5c0543629650cb4c3e7fa899 2023-04-03 02:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:50,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b63531d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:50,729 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:50,729 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,732 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:50,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:50,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:50,733 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,733 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:50,735 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:50,735 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,735 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:50,735 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/61211523fd62432986ff5ef2c7e3e124 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/61211523fd62432986ff5ef2c7e3e124 2023-04-03 02:53:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:50,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:50,746 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:50,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:50,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:50,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:50,746 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:50,748 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:50,748 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,748 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:50,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:50,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,749 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,756 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:50,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:50,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:50,756 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,756 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,756 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,756 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,756 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,756 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:50,756 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:50,756 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:50,756 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,757 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:50,757 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:50,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:50,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:50,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/61211523fd62432986ff5ef2c7e3e124, entries=250, sequenceid=284, filesize=16.7 K 2023-04-03 02:53:50,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/7db6361401e14aca854714822011a9a7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7db6361401e14aca854714822011a9a7 2023-04-03 02:53:50,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7db6361401e14aca854714822011a9a7, entries=150, sequenceid=284, filesize=11.9 K 2023-04-03 02:53:50,790 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/477f50be5c0543629650cb4c3e7fa899 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477f50be5c0543629650cb4c3e7fa899 2023-04-03 02:53:50,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477f50be5c0543629650cb4c3e7fa899, entries=150, sequenceid=284, filesize=11.9 K 2023-04-03 02:53:50,803 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=93.93 KB/96180 for 459f839097be2d0597a0a35d5312fdc4 in 321ms, sequenceid=284, compaction requested=true 2023-04-03 02:53:50,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:50,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:53:50,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-03 02:53:50,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-03 02:53:50,856 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:50,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:50,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:50,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:50,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:50,869 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:50,869 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:50,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:50,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:50,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,872 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:50,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:50,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:50,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:50,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:50,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:50,874 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:50,880 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:50,880 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:50,880 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:50,880 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:50,880 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 02:53:50,880 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,880 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:50,884 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:50,884 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:50,884 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:50,884 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:50,885 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:50,885 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:50,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:50,894 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:50,933 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/27f17e3b4bef4e478b338b5c23b7d537 2023-04-03 02:53:50,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490490927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490490927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490490931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490490934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:50,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490490934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:50,991 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/9794d89cf4b54ddf807df2c1f61d8bb7 2023-04-03 02:53:51,043 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/1e4c6968f7ed4f2689bd99e908db1fe8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/1e4c6968f7ed4f2689bd99e908db1fe8 2023-04-03 02:53:51,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490491040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490491041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,063 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/26c9257ecb274c21918c782f09da35e2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/26c9257ecb274c21918c782f09da35e2 2023-04-03 02:53:51,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490491045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490491050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,065 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/A of 459f839097be2d0597a0a35d5312fdc4 into 1e4c6968f7ed4f2689bd99e908db1fe8(size=12.1 K), total size for store is 28.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:51,066 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:51,066 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/A, priority=12, startTime=1680490430479; duration=0sec 2023-04-03 02:53:51,067 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-03 02:53:51,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490491060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,067 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:53:51,071 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58139 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:53:51,071 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/C is initiating minor compaction (all files) 2023-04-03 02:53:51,071 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/C in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:51,071 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/ea9bb2adc9d8464899dd80b7e0d47ef0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/96251e9e770643eab525d12986b439cf, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3884fcd1c157430d9ba21117436d657d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/268736ad52b54e8f8f17019c4a3d2ba0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477f50be5c0543629650cb4c3e7fa899] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=56.8 K 2023-04-03 02:53:51,072 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting ea9bb2adc9d8464899dd80b7e0d47ef0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680490428414 2023-04-03 02:53:51,072 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 96251e9e770643eab525d12986b439cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680490428463 2023-04-03 02:53:51,073 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 3884fcd1c157430d9ba21117436d657d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680490429134 2023-04-03 02:53:51,074 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 268736ad52b54e8f8f17019c4a3d2ba0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680490430295 2023-04-03 02:53:51,075 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 477f50be5c0543629650cb4c3e7fa899, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680490430397 2023-04-03 02:53:51,084 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/B of 459f839097be2d0597a0a35d5312fdc4 into 26c9257ecb274c21918c782f09da35e2(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:51,085 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:51,086 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/B, priority=12, startTime=1680490430479; duration=0sec 2023-04-03 02:53:51,086 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-03 02:53:51,087 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 5 compacting, 0 eligible, 16 blocking 2023-04-03 02:53:51,087 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 02:53:51,087 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 02:53:51,087 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. because compaction request was cancelled 2023-04-03 02:53:51,087 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-03 02:53:51,088 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 02:53:51,088 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 02:53:51,088 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. because compaction request was cancelled 2023-04-03 02:53:51,089 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-03 02:53:51,090 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 02:53:51,090 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 02:53:51,090 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. because compaction request was cancelled 2023-04-03 02:53:51,099 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#C#compaction#29 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:51,140 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/846ee0010ad34d66b92db368c691b2dd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/846ee0010ad34d66b92db368c691b2dd 2023-04-03 02:53:51,151 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/C of 459f839097be2d0597a0a35d5312fdc4 into 846ee0010ad34d66b92db368c691b2dd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:51,151 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:51,151 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/C, priority=11, startTime=1680490430803; duration=0sec 2023-04-03 02:53:51,152 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:51,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490491259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490491261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490491266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490491267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490491270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,465 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/a9f94bd60d54437090c6103fdf8c1b42 2023-04-03 02:53:51,473 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/27f17e3b4bef4e478b338b5c23b7d537 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/27f17e3b4bef4e478b338b5c23b7d537 2023-04-03 02:53:51,485 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/27f17e3b4bef4e478b338b5c23b7d537, entries=150, sequenceid=301, filesize=11.9 K 2023-04-03 02:53:51,487 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/9794d89cf4b54ddf807df2c1f61d8bb7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/9794d89cf4b54ddf807df2c1f61d8bb7 2023-04-03 02:53:51,496 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/9794d89cf4b54ddf807df2c1f61d8bb7, entries=150, sequenceid=301, filesize=11.9 K 2023-04-03 02:53:51,500 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/a9f94bd60d54437090c6103fdf8c1b42 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a9f94bd60d54437090c6103fdf8c1b42 2023-04-03 02:53:51,512 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a9f94bd60d54437090c6103fdf8c1b42, entries=150, sequenceid=301, filesize=11.9 K 2023-04-03 02:53:51,517 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 459f839097be2d0597a0a35d5312fdc4 in 637ms, sequenceid=301, compaction requested=true 2023-04-03 02:53:51,517 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:51,517 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:51,518 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:51,518 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:51,519 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,519 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:51,519 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,525 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,525 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453053a3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:51,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:51,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,530 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,530 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:51,530 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:51,530 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:51,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,532 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:51,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:51,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:51,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:51,540 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:51,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:51,547 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:51,547 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:51,547 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:51,547 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:51,548 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:51,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,550 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,550 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:51,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:51,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,553 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:51,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:51,557 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:51,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:51,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:51,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:51,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:51,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:51,559 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,559 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,559 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,559 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:51,559 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:51,559 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:51,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:51,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 02:53:51,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:51,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:51,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:51,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:51,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:51,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:51,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490491586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490491584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490491590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490491592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490491594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,622 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/a96519779818495889cda5d0aee88da1 2023-04-03 02:53:51,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/064538efcd5748b7a0142381d63241d6 2023-04-03 02:53:51,659 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:51,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:51,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:51,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:51,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:51,667 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:51,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:51,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:51,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:51,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:51,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,671 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:51,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:51,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:51,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:51,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:51,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:51,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:51,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:51,677 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:51,677 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:51,677 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:51,677 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:51,681 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:51,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:51,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a14b3b9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:51,684 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:51,684 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,685 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:51,686 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:51,686 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:51,686 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,686 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:51,687 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:51,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:51,691 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:51,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:51,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:51,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:51,691 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:51,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490491698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490491698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,701 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,701 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:51,701 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:51,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:51,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,704 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:51,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490491711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490491713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,718 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,718 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:51,718 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:51,718 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:51,718 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,719 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:51,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:51,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:51,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:51,719 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:51,722 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:51,722 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:51,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:51,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490491714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,719 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,725 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:51,725 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/45587ca221504628818f39ba905f60ec 2023-04-03 02:53:51,736 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/a96519779818495889cda5d0aee88da1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a96519779818495889cda5d0aee88da1 2023-04-03 02:53:51,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a96519779818495889cda5d0aee88da1, entries=200, sequenceid=320, filesize=14.3 K 2023-04-03 02:53:51,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/064538efcd5748b7a0142381d63241d6 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/064538efcd5748b7a0142381d63241d6 2023-04-03 02:53:51,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/064538efcd5748b7a0142381d63241d6, entries=150, sequenceid=320, filesize=11.9 K 2023-04-03 02:53:51,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/45587ca221504628818f39ba905f60ec as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/45587ca221504628818f39ba905f60ec 2023-04-03 02:53:51,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/45587ca221504628818f39ba905f60ec, entries=150, sequenceid=320, filesize=11.9 K 2023-04-03 02:53:51,790 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 459f839097be2d0597a0a35d5312fdc4 in 224ms, sequenceid=320, compaction requested=true 2023-04-03 02:53:51,790 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:51,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:51,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:51,790 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:53:51,790 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:53:51,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:51,797 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56254 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:53:51,797 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/A is initiating minor compaction (all files) 2023-04-03 02:53:51,798 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/A in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:51,798 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/1e4c6968f7ed4f2689bd99e908db1fe8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/61211523fd62432986ff5ef2c7e3e124, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/27f17e3b4bef4e478b338b5c23b7d537, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a96519779818495889cda5d0aee88da1] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=54.9 K 2023-04-03 02:53:51,798 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 1e4c6968f7ed4f2689bd99e908db1fe8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680490429134 2023-04-03 02:53:51,800 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48934 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:53:51,801 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/B is initiating minor compaction (all files) 2023-04-03 02:53:51,801 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/B in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:51,801 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/26c9257ecb274c21918c782f09da35e2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7db6361401e14aca854714822011a9a7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/9794d89cf4b54ddf807df2c1f61d8bb7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/064538efcd5748b7a0142381d63241d6] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=47.8 K 2023-04-03 02:53:51,801 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 61211523fd62432986ff5ef2c7e3e124, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680490430347 2023-04-03 02:53:51,802 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 26c9257ecb274c21918c782f09da35e2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680490429134 2023-04-03 02:53:51,802 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 27f17e3b4bef4e478b338b5c23b7d537, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680490430537 2023-04-03 02:53:51,823 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7db6361401e14aca854714822011a9a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680490430397 2023-04-03 02:53:51,823 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:51,823 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a96519779818495889cda5d0aee88da1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680490430926 2023-04-03 02:53:51,824 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 9794d89cf4b54ddf807df2c1f61d8bb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680490430537 2023-04-03 02:53:51,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:51,825 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 064538efcd5748b7a0142381d63241d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680490430926 2023-04-03 02:53:51,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:51,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:51,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:51,833 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:51,833 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:51,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:51,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:51,835 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,835 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:51,837 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:51,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:51,838 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,838 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:51,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:51,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:51,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:51,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:51,845 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:51,845 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:51,845 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:51,847 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:51,847 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:51,850 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 02:53:51,850 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:51,850 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:51,851 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:51,851 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:51,851 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:51,851 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:51,857 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#B#compaction#33 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:51,862 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#A#compaction#34 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:51,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:51,906 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:51,914 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/d0b6d445fb0643c8aa8214e827b5011a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d0b6d445fb0643c8aa8214e827b5011a 2023-04-03 02:53:51,921 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/7ffafcf7eb464b1d846203a0ce50b041 2023-04-03 02:53:51,922 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/cf320548934341e9bd4c1dd23b6b0ced as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/cf320548934341e9bd4c1dd23b6b0ced 2023-04-03 02:53:51,926 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/B of 459f839097be2d0597a0a35d5312fdc4 into d0b6d445fb0643c8aa8214e827b5011a(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:51,926 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:51,926 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/B, priority=12, startTime=1680490431790; duration=0sec 2023-04-03 02:53:51,927 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:51,927 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:53:51,938 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36832 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:53:51,938 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/C is initiating minor compaction (all files) 2023-04-03 02:53:51,938 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/C in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:51,938 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/846ee0010ad34d66b92db368c691b2dd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a9f94bd60d54437090c6103fdf8c1b42, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/45587ca221504628818f39ba905f60ec] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=36.0 K 2023-04-03 02:53:51,939 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 846ee0010ad34d66b92db368c691b2dd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680490430397 2023-04-03 02:53:51,940 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a9f94bd60d54437090c6103fdf8c1b42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680490430537 2023-04-03 02:53:51,941 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 45587ca221504628818f39ba905f60ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680490430926 2023-04-03 02:53:51,955 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/A of 459f839097be2d0597a0a35d5312fdc4 into cf320548934341e9bd4c1dd23b6b0ced(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:51,961 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:51,961 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/A, priority=12, startTime=1680490431790; duration=0sec 2023-04-03 02:53:51,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,961 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:51,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490491947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,962 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#C#compaction#37 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:51,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490491953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490491955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:51,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490491956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490491961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:51,974 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/a714f56a0fbf413292acdc8bde4d91b7 2023-04-03 02:53:52,007 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/db13386f44ce4988b7657395b8ef482d 2023-04-03 02:53:52,015 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/7ffafcf7eb464b1d846203a0ce50b041 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/7ffafcf7eb464b1d846203a0ce50b041 2023-04-03 02:53:52,023 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/7ffafcf7eb464b1d846203a0ce50b041, entries=150, sequenceid=336, filesize=11.9 K 2023-04-03 02:53:52,028 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/a714f56a0fbf413292acdc8bde4d91b7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/a714f56a0fbf413292acdc8bde4d91b7 2023-04-03 02:53:52,035 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/a714f56a0fbf413292acdc8bde4d91b7, entries=150, sequenceid=336, filesize=11.9 K 2023-04-03 02:53:52,037 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/db13386f44ce4988b7657395b8ef482d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/db13386f44ce4988b7657395b8ef482d 2023-04-03 02:53:52,044 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/db13386f44ce4988b7657395b8ef482d, entries=150, sequenceid=336, filesize=11.9 K 2023-04-03 02:53:52,050 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 459f839097be2d0597a0a35d5312fdc4 in 200ms, sequenceid=336, compaction requested=false 2023-04-03 02:53:52,051 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:52,051 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,055 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,057 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5770daab[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:52,059 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:52,059 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:52,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:52,064 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:52,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 02:53:52,067 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:52,067 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:52,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:52,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:52,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:52,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:52,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:52,070 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:52,071 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:52,071 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:52,071 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:52,071 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:52,073 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:52,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,077 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,080 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,087 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:52,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:52,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:52,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:52,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,088 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,088 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,088 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:52,088 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:52,088 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:52,088 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:52,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:52,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:52,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490492089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490492090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47586 deadline: 1680490492091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490492094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490492095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,119 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/ba50c01bc97347e09f0c0afc1fb8c8b7 2023-04-03 02:53:52,135 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/efb1f87d9f4b498488ef024932118d4e 2023-04-03 02:53:52,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/28dba09954784eeb923581296ea60bd8 2023-04-03 02:53:52,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/ba50c01bc97347e09f0c0afc1fb8c8b7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ba50c01bc97347e09f0c0afc1fb8c8b7 2023-04-03 02:53:52,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ba50c01bc97347e09f0c0afc1fb8c8b7, entries=150, sequenceid=356, filesize=11.9 K 2023-04-03 02:53:52,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/efb1f87d9f4b498488ef024932118d4e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/efb1f87d9f4b498488ef024932118d4e 2023-04-03 02:53:52,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/efb1f87d9f4b498488ef024932118d4e, entries=150, sequenceid=356, filesize=11.9 K 2023-04-03 02:53:52,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/28dba09954784eeb923581296ea60bd8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/28dba09954784eeb923581296ea60bd8 2023-04-03 02:53:52,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/28dba09954784eeb923581296ea60bd8, entries=150, sequenceid=356, filesize=11.9 K 2023-04-03 02:53:52,188 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:52,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:52,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 459f839097be2d0597a0a35d5312fdc4 in 122ms, sequenceid=356, compaction requested=true 2023-04-03 02:53:52,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:52,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:52,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:52,190 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:53:52,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:53:52,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:52,193 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:53:52,193 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/A is initiating minor compaction (all files) 2023-04-03 02:53:52,193 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/A in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,193 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/cf320548934341e9bd4c1dd23b6b0ced, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/7ffafcf7eb464b1d846203a0ce50b041, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ba50c01bc97347e09f0c0afc1fb8c8b7] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=36.1 K 2023-04-03 02:53:52,194 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting cf320548934341e9bd4c1dd23b6b0ced, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680490430926 2023-04-03 02:53:52,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:52,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:52,197 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:52,197 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:52,198 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:52,198 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:52,199 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,199 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,199 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 7ffafcf7eb464b1d846203a0ce50b041, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680490431584 2023-04-03 02:53:52,199 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting ba50c01bc97347e09f0c0afc1fb8c8b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680490431952 2023-04-03 02:53:52,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:52,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:52,200 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 02:53:52,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:52,200 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:52,201 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,201 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:52,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:52,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:52,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:52,203 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:52,203 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:52,204 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:52,205 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,205 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:52,205 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:52,205 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:52,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:52,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,209 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,209 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,209 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:52,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da90e3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:52,211 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:52,211 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,216 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,217 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:52,217 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:52,217 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:52,217 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,217 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:52,220 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#A#compaction#43 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:52,221 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,222 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:52,222 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:52,222 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490492222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490492222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:52,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:52,229 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:52,229 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:52,229 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:52,229 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:52,229 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:52,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490492224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,231 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490492224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490492225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,231 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,232 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:52,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:52,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,235 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:52,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:52,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:52,244 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:52,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:52,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:52,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,245 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:52,245 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:52,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:52,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:52,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490492329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490492329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490492334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490492334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490492335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,345 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:52,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:52,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:52,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:52,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:52,349 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:52,349 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:52,350 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:52,350 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:52,351 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,351 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,353 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:52,353 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:52,354 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,354 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:52,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:52,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:52,355 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:52,355 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:52,356 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,356 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:52,356 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:52,357 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:52,358 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,358 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:52,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413076d0[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:52,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:52,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,366 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,366 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:52,366 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:52,366 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,367 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:52,366 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:52,368 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:52,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:52,373 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:52,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:52,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:52,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:52,373 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:52,376 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,376 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,376 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:52,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,377 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,379 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:52,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:52,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:52,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:52,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,382 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:52,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:52,382 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:52,383 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:52,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:52,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:52,400 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/5e55053cc5414204aba6c71b8039f747 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/5e55053cc5414204aba6c71b8039f747 2023-04-03 02:53:52,415 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/C of 459f839097be2d0597a0a35d5312fdc4 into 5e55053cc5414204aba6c71b8039f747(size=12.3 K), total size for store is 36.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:52,415 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:52,415 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/C, priority=13, startTime=1680490431790; duration=0sec 2023-04-03 02:53:52,416 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:53:52,416 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:53:52,417 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:53:52,418 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/B is initiating minor compaction (all files) 2023-04-03 02:53:52,418 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/B in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,418 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d0b6d445fb0643c8aa8214e827b5011a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/a714f56a0fbf413292acdc8bde4d91b7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/efb1f87d9f4b498488ef024932118d4e] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=36.1 K 2023-04-03 02:53:52,419 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d0b6d445fb0643c8aa8214e827b5011a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680490430926 2023-04-03 02:53:52,420 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a714f56a0fbf413292acdc8bde4d91b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680490431584 2023-04-03 02:53:52,420 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting efb1f87d9f4b498488ef024932118d4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680490431952 2023-04-03 02:53:52,429 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#B#compaction#44 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:52,484 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:52,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:52,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:52,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:52,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:52,489 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:52,489 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:52,490 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:52,490 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:52,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,492 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:52,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:52,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:52,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:52,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:52,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:52,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:52,496 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,496 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:52,496 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:52,496 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:52,498 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,498 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,498 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,498 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e768c34[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:52,500 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:52,500 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,502 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:52,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:52,502 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:52,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:52,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:52,509 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:52,509 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:52,509 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:52,509 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:52,509 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:52,510 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,510 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,510 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:52,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,511 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,513 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:52,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:52,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:52,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:52,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:52,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:52,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:52,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,516 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:52,517 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,517 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,517 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,517 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:52,517 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490492534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490492535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490492538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490492538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490492539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,617 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:52,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:52,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:52,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:52,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:52,622 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:52,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:52,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:52,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:52,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,626 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,626 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:52,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:52,627 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,627 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:52,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:52,630 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,630 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:52,630 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:52,630 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:52,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:52,631 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,632 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f02bad3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:52,634 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:52,634 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,635 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:52,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:52,635 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,635 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:52,637 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,637 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,637 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:52,637 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,640 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/bd6cde21a80443a18530a4a76930e484 2023-04-03 02:53:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:52,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:52,642 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:52,642 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:52,642 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:52,642 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:52,642 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:52,648 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,648 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:52,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,648 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:52,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:52,651 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,654 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,659 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:52,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:52,660 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,660 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:52,660 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:52,660 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,660 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,660 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:52,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:52,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:52,660 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:52,661 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:52,666 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:52,660 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:52,666 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,667 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/adec6bc86a5a4839985a6a3acd9db52d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/adec6bc86a5a4839985a6a3acd9db52d 2023-04-03 02:53:52,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/5eaf3075220841edaf559be32e9aba27 2023-04-03 02:53:52,675 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/A of 459f839097be2d0597a0a35d5312fdc4 into adec6bc86a5a4839985a6a3acd9db52d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:52,675 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:52,677 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/A, priority=13, startTime=1680490432190; duration=0sec 2023-04-03 02:53:52,677 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:52,677 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:53:52,678 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:53:52,679 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/C is initiating minor compaction (all files) 2023-04-03 02:53:52,679 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/C in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,679 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/5e55053cc5414204aba6c71b8039f747, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/db13386f44ce4988b7657395b8ef482d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/28dba09954784eeb923581296ea60bd8] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=36.1 K 2023-04-03 02:53:52,679 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 5e55053cc5414204aba6c71b8039f747, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680490430926 2023-04-03 02:53:52,680 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting db13386f44ce4988b7657395b8ef482d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680490431584 2023-04-03 02:53:52,682 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 28dba09954784eeb923581296ea60bd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680490431952 2023-04-03 02:53:52,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/ef39edd961634ee38aa7cd48df4dcf08 2023-04-03 02:53:52,693 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#C#compaction#47 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:52,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/bd6cde21a80443a18530a4a76930e484 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/bd6cde21a80443a18530a4a76930e484 2023-04-03 02:53:52,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/bd6cde21a80443a18530a4a76930e484, entries=200, sequenceid=371, filesize=14.3 K 2023-04-03 02:53:52,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/5eaf3075220841edaf559be32e9aba27 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/5eaf3075220841edaf559be32e9aba27 2023-04-03 02:53:52,713 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/5eaf3075220841edaf559be32e9aba27, entries=150, sequenceid=371, filesize=11.9 K 2023-04-03 02:53:52,714 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/ef39edd961634ee38aa7cd48df4dcf08 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/ef39edd961634ee38aa7cd48df4dcf08 2023-04-03 02:53:52,725 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/ef39edd961634ee38aa7cd48df4dcf08, entries=150, sequenceid=371, filesize=11.9 K 2023-04-03 02:53:52,725 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 459f839097be2d0597a0a35d5312fdc4 in 525ms, sequenceid=371, compaction requested=false 2023-04-03 02:53:52,726 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:52,766 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:52,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:52,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:52,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:52,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:52,774 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:52,774 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:52,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:52,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:52,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,777 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:52,777 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,777 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,777 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:52,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:52,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:52,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:52,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:52,779 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:52,779 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:52,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:52,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:52,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:52,780 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:52,780 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:52,780 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 02:53:52,780 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:52,780 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:52,780 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:52,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:52,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:52,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:52,801 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/672bc373f29747ad9ac8799be95cb5b5 2023-04-03 02:53:52,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:52,841 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:52,852 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/922cc7f7b3c44156b92687a8a4e72610 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/922cc7f7b3c44156b92687a8a4e72610 2023-04-03 02:53:52,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47670 deadline: 1680490492851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490492853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490492854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,859 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/B of 459f839097be2d0597a0a35d5312fdc4 into 922cc7f7b3c44156b92687a8a4e72610(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:52,859 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:52,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,859 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/B, priority=13, startTime=1680490432190; duration=0sec 2023-04-03 02:53:52,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490492854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,860 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:52,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47654 deadline: 1680490492855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490492957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490492958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490492961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:52,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490492961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:52,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490492961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,118 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/a764a7f1e8ed41d7940ddc551dcb4826 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a764a7f1e8ed41d7940ddc551dcb4826 2023-04-03 02:53:53,126 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/C of 459f839097be2d0597a0a35d5312fdc4 into a764a7f1e8ed41d7940ddc551dcb4826(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:53,126 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:53,126 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/C, priority=13, startTime=1680490432190; duration=0sec 2023-04-03 02:53:53,126 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:53,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490493163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490493164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490493165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47626 deadline: 1680490493166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490493166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,228 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/1d6a2bad49124b73a6f80466fe047cf4 2023-04-03 02:53:53,243 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/2e3bfbeba47a4868ba61853e906e0f51 2023-04-03 02:53:53,251 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/672bc373f29747ad9ac8799be95cb5b5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/672bc373f29747ad9ac8799be95cb5b5 2023-04-03 02:53:53,257 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/672bc373f29747ad9ac8799be95cb5b5, entries=150, sequenceid=390, filesize=11.9 K 2023-04-03 02:53:53,258 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/1d6a2bad49124b73a6f80466fe047cf4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/1d6a2bad49124b73a6f80466fe047cf4 2023-04-03 02:53:53,264 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/1d6a2bad49124b73a6f80466fe047cf4, entries=150, sequenceid=390, filesize=11.9 K 2023-04-03 02:53:53,265 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/2e3bfbeba47a4868ba61853e906e0f51 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/2e3bfbeba47a4868ba61853e906e0f51 2023-04-03 02:53:53,270 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/2e3bfbeba47a4868ba61853e906e0f51, entries=150, sequenceid=390, filesize=11.9 K 2023-04-03 02:53:53,271 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 459f839097be2d0597a0a35d5312fdc4 in 491ms, sequenceid=390, compaction requested=true 2023-04-03 02:53:53,271 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:53,271 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:53,271 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:53,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,275 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:53,276 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,276 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:53,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:53,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d8d6458[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:53,278 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:53,278 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,280 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,281 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:53,281 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:53,281 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:53,281 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,281 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:53,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:53,284 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:53,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:53,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:53,288 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:53,288 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:53,288 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:53,289 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:53,289 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:53,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:53,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:53,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:53,292 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,294 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:53,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:53,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:53,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:53,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:53,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:53,297 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:53,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:53,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:53,297 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:53,299 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:53,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:53,399 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:53,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:53,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:53,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:53,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:53,404 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:53,404 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:53,405 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:53,405 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:53,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:53,407 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,407 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:53,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:53,408 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,408 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:53,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:53,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:53,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:53,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:53,412 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:53,412 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:53,412 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:53,412 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:53,412 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:53,412 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 02:53:53,412 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:53,413 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:53,413 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:53,413 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:53,413 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:53,413 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:53,426 WARN [DataStreamer for file /user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/5311b6d054e444e98025b949d0783a3f] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-03 02:53:53,426 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/5311b6d054e444e98025b949d0783a3f 2023-04-03 02:53:53,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:53,469 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:53,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490493509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490493510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490493511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490493511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490493512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490493614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490493614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490493614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490493632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490493634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490493835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490493836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490493836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490493838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:53,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490493838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,857 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/cfd82e03e3604f41929cf7b0e10848e7 2023-04-03 02:53:53,875 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/4319afea2ee346c0a0d8ad34d549101f 2023-04-03 02:53:53,881 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/5311b6d054e444e98025b949d0783a3f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5311b6d054e444e98025b949d0783a3f 2023-04-03 02:53:53,893 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5311b6d054e444e98025b949d0783a3f, entries=150, sequenceid=407, filesize=11.9 K 2023-04-03 02:53:53,895 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/cfd82e03e3604f41929cf7b0e10848e7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/cfd82e03e3604f41929cf7b0e10848e7 2023-04-03 02:53:53,901 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/cfd82e03e3604f41929cf7b0e10848e7, entries=150, sequenceid=407, filesize=11.9 K 2023-04-03 02:53:53,902 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/4319afea2ee346c0a0d8ad34d549101f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/4319afea2ee346c0a0d8ad34d549101f 2023-04-03 02:53:53,913 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/4319afea2ee346c0a0d8ad34d549101f, entries=150, sequenceid=407, filesize=11.9 K 2023-04-03 02:53:53,914 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 459f839097be2d0597a0a35d5312fdc4 in 502ms, sequenceid=407, compaction requested=true 2023-04-03 02:53:53,914 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:53,914 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:53,914 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:53,915 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:53,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:53,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:53,918 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,918 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:53,918 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,918 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:53,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:53,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fc905cf[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:53,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:53,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,921 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,922 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:53,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:53,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:53,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:53,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:53,928 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:53,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:53,931 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:53,931 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:53,931 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:53,931 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:53,931 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:53,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:53,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:53,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,936 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:53,941 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:53,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:53,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:53,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:53,945 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:53,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:53,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:53,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:53,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:53,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:53,945 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:53,945 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:53,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:53,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:54,045 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:54,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:54,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:54,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:54,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:54,050 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:54,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:54,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:54,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:54,051 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:54,051 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:54,052 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,052 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:54,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:54,053 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,053 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:54,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:54,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:54,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:54,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:54,056 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:54,056 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:54,056 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 02:53:54,057 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:54,057 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:54,057 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:54,057 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:54,058 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:54,058 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:54,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:54,141 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:54,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490494151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490494151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47642 deadline: 1680490494155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490494158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490494159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490494263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490494264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490494263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490494265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490494265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490494466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490494466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490494467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490494467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,477 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/e7c5d0a716114315a1dbd0746d6993f7 2023-04-03 02:53:54,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490494476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490494772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490494774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490494775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490494775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:54,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490494779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,898 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/eefb5b77d22b4edc875aeaf0ae832d03 2023-04-03 02:53:54,936 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/6fa17bcf9e114c4d849f8b25ae9d7e56 2023-04-03 02:53:54,952 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/e7c5d0a716114315a1dbd0746d6993f7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/e7c5d0a716114315a1dbd0746d6993f7 2023-04-03 02:53:54,961 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/e7c5d0a716114315a1dbd0746d6993f7, entries=150, sequenceid=425, filesize=11.9 K 2023-04-03 02:53:54,963 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/eefb5b77d22b4edc875aeaf0ae832d03 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/eefb5b77d22b4edc875aeaf0ae832d03 2023-04-03 02:53:54,969 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/eefb5b77d22b4edc875aeaf0ae832d03, entries=150, sequenceid=425, filesize=11.9 K 2023-04-03 02:53:54,970 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/6fa17bcf9e114c4d849f8b25ae9d7e56 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6fa17bcf9e114c4d849f8b25ae9d7e56 2023-04-03 02:53:54,979 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6fa17bcf9e114c4d849f8b25ae9d7e56, entries=150, sequenceid=425, filesize=11.9 K 2023-04-03 02:53:54,980 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 459f839097be2d0597a0a35d5312fdc4 in 924ms, sequenceid=425, compaction requested=true 2023-04-03 02:53:54,980 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:54,980 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:54,980 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:54,987 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:54,987 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:54,990 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:54,990 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75e4372e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:54,995 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:54,995 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:54,996 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:54,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:54,997 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:54,997 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:55,000 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:55,003 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:55,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:55,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:55,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:55,003 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:55,007 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,008 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,009 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,008 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,012 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:55,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,020 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:55,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:55,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:55,020 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:55,020 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:55,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,121 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:55,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:55,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:55,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:55,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:55,125 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:55,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:55,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:55,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:55,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:55,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:55,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,128 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:55,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:55,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:55,129 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:55,129 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:55,129 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:55,129 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:55,129 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:55,129 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:55,129 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:55,130 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 02:53:55,130 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:55,130 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:55,130 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:55,130 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:55,130 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:55,130 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:55,145 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/c0b589a652ec46f981a03287de16dc8a 2023-04-03 02:53:55,193 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/813f4ed329284119adcfb9a4d49aa910 2023-04-03 02:53:55,219 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/b9291a29d2b647679f89771923cb0f5f 2023-04-03 02:53:55,225 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/c0b589a652ec46f981a03287de16dc8a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/c0b589a652ec46f981a03287de16dc8a 2023-04-03 02:53:55,230 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/c0b589a652ec46f981a03287de16dc8a, entries=150, sequenceid=439, filesize=11.9 K 2023-04-03 02:53:55,231 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/813f4ed329284119adcfb9a4d49aa910 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/813f4ed329284119adcfb9a4d49aa910 2023-04-03 02:53:55,237 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/813f4ed329284119adcfb9a4d49aa910, entries=150, sequenceid=439, filesize=11.9 K 2023-04-03 02:53:55,239 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/b9291a29d2b647679f89771923cb0f5f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/b9291a29d2b647679f89771923cb0f5f 2023-04-03 02:53:55,245 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/b9291a29d2b647679f89771923cb0f5f, entries=150, sequenceid=439, filesize=11.9 K 2023-04-03 02:53:55,246 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 459f839097be2d0597a0a35d5312fdc4 in 116ms, sequenceid=439, compaction requested=true 2023-04-03 02:53:55,246 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:55,246 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:55,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:55,247 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:55,247 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,247 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:55,247 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:55,249 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7acf63[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:55,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:55,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,256 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:55,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:55,257 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:55,258 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:55,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:55,261 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:55,262 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:55,262 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:55,262 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:55,262 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:55,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,265 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,277 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:55,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:55,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:55,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:55,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:55,282 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:55,282 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:55,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,282 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:55,288 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:53:55,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:55,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:55,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:55,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:55,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:55,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:55,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680490435322, max timestamp=1680490435335Num uniques 859; 2023-04-03 02:53:55,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-03 02:53:55,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680490435322, max timestamp=1680490435335Num uniques 859; 2023-04-03 02:53:55,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680490435322, max timestamp=1680490435335Num uniques 859; 2023-04-03 02:53:55,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-03 02:53:55,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-03 02:53:55,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490495350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490495364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490495365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490495365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490495368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,382 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:55,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:55,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:55,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:55,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:55,387 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:55,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:55,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:55,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:55,388 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,388 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,391 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:55,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:55,391 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:55,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:55,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:55,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:55,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:55,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:55,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:55,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:55,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:55,398 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:55,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:55,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78e2151d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:55,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:55,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:55,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,401 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,401 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,402 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:55,401 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:55,403 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:55,408 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:55,408 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:55,408 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:55,408 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:55,408 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:55,411 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,413 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,418 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,418 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,418 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,418 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:55,418 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,418 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:55,418 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:55,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:55,419 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:55,418 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,419 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:55,419 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,419 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490495466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490495472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490495473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490495474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490495474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,519 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:55,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:55,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:55,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:55,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:55,528 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:55,528 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:55,528 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:55,528 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:55,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,530 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:55,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:55,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:55,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:55,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:55,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:55,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:55,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:55,553 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:55,553 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:55,553 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:55,553 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:55,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5933c370[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:55,571 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:55,571 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,574 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:55,574 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:55,574 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,574 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:55,574 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,576 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:55,584 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:55,584 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:55,584 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:55,585 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:55,585 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:55,587 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,587 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,588 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,590 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,592 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,599 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:55,599 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:55,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,600 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,600 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,600 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,600 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,600 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:55,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:55,600 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,600 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,600 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:55,601 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:55,601 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:55,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490495672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490495679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490495679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490495680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490495680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,701 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:55,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:55,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/f6f7b054a9fd4ca8adf4167670c8de51 2023-04-03 02:53:55,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:55,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:55,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:55,706 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:55,706 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:55,706 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:55,706 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:55,707 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,707 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,708 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:55,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:55,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:55,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:55,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:55,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:55,727 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:55,727 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:55,728 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:55,728 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:55,728 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:55,728 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,731 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,731 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,731 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:55,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ed342e4[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:55,734 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:55,735 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,737 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,738 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,738 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:55,738 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:55,738 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:55,738 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:55,739 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:55,739 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,739 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:55,739 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:55,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:55,745 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:55,745 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:55,745 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:55,745 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:55,745 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:55,747 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,747 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,748 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,748 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,750 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,752 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:55,752 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:55,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,753 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:55,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:55,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,754 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,754 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,754 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,754 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:55,754 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:55,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:55,854 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:55,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:55,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:55,880 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:55,880 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:55,880 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:55,880 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:55,881 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,881 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,883 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:55,883 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:55,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:55,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:55,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:55,896 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:55,896 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:55,896 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:55,906 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:55,906 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:55,908 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:55,908 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:55,909 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:55,909 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:55,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:55,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:55,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:55,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:55,911 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,911 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:55,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ada4357[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:55,914 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:55,914 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,916 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:55,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:55,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:55,924 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:55,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:55,925 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:55,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:55,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:55,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:55,925 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:55,928 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,928 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,929 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:55,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,932 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,934 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:55,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:55,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:55,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:55,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,937 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:55,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:55,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:55,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:55,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:55,938 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:55,938 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:55,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:55,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490495978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490495987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490495987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:55,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:55,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490495988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490495991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,038 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:56,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:56,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:56,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:56,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:56,043 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:56,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:56,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:56,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:56,044 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:56,044 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:56,045 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:56,045 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:56,045 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,045 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:56,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:56,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:56,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:56,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:56,047 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:56,047 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:56,047 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:56,047 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:56,050 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@633284b8[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:56,052 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:56,052 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:56,054 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:56,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:56,058 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,058 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:56,064 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:56,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:56,067 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:56,067 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:56,067 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:56,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:56,068 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:56,071 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:56,071 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:56,071 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:56,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:56,072 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:56,073 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:56,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:56,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:56,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:56,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:56,077 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:56,077 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:56,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:56,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:56,077 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,077 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:56,077 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:56,077 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:56,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:56,077 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:56,077 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:56,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:56,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/db8ca8b74f9b471ea434c7b89f082ce3 2023-04-03 02:53:56,153 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/91bfee7fd70f4a3b8a39d0bc9c00bc83 2023-04-03 02:53:56,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/f6f7b054a9fd4ca8adf4167670c8de51 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f6f7b054a9fd4ca8adf4167670c8de51 2023-04-03 02:53:56,165 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f6f7b054a9fd4ca8adf4167670c8de51, entries=350, sequenceid=449, filesize=21.4 K 2023-04-03 02:53:56,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/db8ca8b74f9b471ea434c7b89f082ce3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/db8ca8b74f9b471ea434c7b89f082ce3 2023-04-03 02:53:56,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/db8ca8b74f9b471ea434c7b89f082ce3, entries=150, sequenceid=449, filesize=11.9 K 2023-04-03 02:53:56,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/91bfee7fd70f4a3b8a39d0bc9c00bc83 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/91bfee7fd70f4a3b8a39d0bc9c00bc83 2023-04-03 02:53:56,177 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:56,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:56,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/91bfee7fd70f4a3b8a39d0bc9c00bc83, entries=150, sequenceid=449, filesize=11.9 K 2023-04-03 02:53:56,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:56,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:56,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:56,182 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:56,182 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:56,182 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:56,182 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:56,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 459f839097be2d0597a0a35d5312fdc4 in 894ms, sequenceid=449, compaction requested=true 2023-04-03 02:53:56,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:56,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:56,182 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-03 02:53:56,182 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-03 02:53:56,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:56,182 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:56,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:56,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:56,185 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:56,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:56,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:56,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:56,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:56,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:56,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:56,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:56,187 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 97984 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-03 02:53:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:56,187 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/A is initiating minor compaction (all files) 2023-04-03 02:53:56,187 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/A in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:56,188 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/adec6bc86a5a4839985a6a3acd9db52d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/bd6cde21a80443a18530a4a76930e484, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/672bc373f29747ad9ac8799be95cb5b5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5311b6d054e444e98025b949d0783a3f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/e7c5d0a716114315a1dbd0746d6993f7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/c0b589a652ec46f981a03287de16dc8a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f6f7b054a9fd4ca8adf4167670c8de51] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=95.7 K 2023-04-03 02:53:56,188 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85780 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-03 02:53:56,188 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting adec6bc86a5a4839985a6a3acd9db52d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680490431952 2023-04-03 02:53:56,191 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:56,193 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:56,193 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:56,188 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/B is initiating minor compaction (all files) 2023-04-03 02:53:56,193 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting bd6cde21a80443a18530a4a76930e484, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680490432070 2023-04-03 02:53:56,193 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/B in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:56,193 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:56,193 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/922cc7f7b3c44156b92687a8a4e72610, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/5eaf3075220841edaf559be32e9aba27, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/1d6a2bad49124b73a6f80466fe047cf4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/cfd82e03e3604f41929cf7b0e10848e7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/eefb5b77d22b4edc875aeaf0ae832d03, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/813f4ed329284119adcfb9a4d49aa910, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/db8ca8b74f9b471ea434c7b89f082ce3] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=83.8 K 2023-04-03 02:53:56,193 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=147.60 KB heapSize=401.39 KB 2023-04-03 02:53:56,194 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:56,194 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:56,194 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:56,194 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:56,195 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 922cc7f7b3c44156b92687a8a4e72610, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680490431952 2023-04-03 02:53:56,197 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:56,194 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 672bc373f29747ad9ac8799be95cb5b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680490432220 2023-04-03 02:53:56,198 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:56,198 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 5eaf3075220841edaf559be32e9aba27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680490432070 2023-04-03 02:53:56,198 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 5311b6d054e444e98025b949d0783a3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680490432851 2023-04-03 02:53:56,202 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 1d6a2bad49124b73a6f80466fe047cf4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680490432220 2023-04-03 02:53:56,202 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e7c5d0a716114315a1dbd0746d6993f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680490433508 2023-04-03 02:53:56,203 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting cfd82e03e3604f41929cf7b0e10848e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680490432851 2023-04-03 02:53:56,203 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c0b589a652ec46f981a03287de16dc8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680490434147 2023-04-03 02:53:56,203 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting eefb5b77d22b4edc875aeaf0ae832d03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680490433508 2023-04-03 02:53:56,203 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting f6f7b054a9fd4ca8adf4167670c8de51, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680490435278 2023-04-03 02:53:56,204 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 813f4ed329284119adcfb9a4d49aa910, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680490434147 2023-04-03 02:53:56,204 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting db8ca8b74f9b471ea434c7b89f082ce3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680490435283 2023-04-03 02:53:56,223 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#A#compaction#64 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:56,230 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#B#compaction#65 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:56,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:56,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:56,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490496501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490496503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490496503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490496504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490496505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490496607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490496607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490496610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490496610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,632 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/3437d51aecad42dda50078e8dc47c1c0 2023-04-03 02:53:56,641 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/e6596c3216174329aacd0121b9cbb2ab as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/e6596c3216174329aacd0121b9cbb2ab 2023-04-03 02:53:56,647 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/802683618e5445eb9bcb972c2568661e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/802683618e5445eb9bcb972c2568661e 2023-04-03 02:53:56,650 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/A of 459f839097be2d0597a0a35d5312fdc4 into e6596c3216174329aacd0121b9cbb2ab(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:56,650 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:56,650 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/A, priority=9, startTime=1680490436182; duration=0sec 2023-04-03 02:53:56,650 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:56,650 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-03 02:53:56,655 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/B of 459f839097be2d0597a0a35d5312fdc4 into 802683618e5445eb9bcb972c2568661e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:56,655 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:56,655 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/B, priority=9, startTime=1680490436182; duration=0sec 2023-04-03 02:53:56,655 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:56,656 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/fc0c7bb99ae240b2987d9b13f3ca0b6b 2023-04-03 02:53:56,656 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85780 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-03 02:53:56,656 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/C is initiating minor compaction (all files) 2023-04-03 02:53:56,656 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/C in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:56,657 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a764a7f1e8ed41d7940ddc551dcb4826, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/ef39edd961634ee38aa7cd48df4dcf08, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/2e3bfbeba47a4868ba61853e906e0f51, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/4319afea2ee346c0a0d8ad34d549101f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6fa17bcf9e114c4d849f8b25ae9d7e56, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/b9291a29d2b647679f89771923cb0f5f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/91bfee7fd70f4a3b8a39d0bc9c00bc83] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=83.8 K 2023-04-03 02:53:56,657 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a764a7f1e8ed41d7940ddc551dcb4826, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680490431952 2023-04-03 02:53:56,658 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting ef39edd961634ee38aa7cd48df4dcf08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680490432070 2023-04-03 02:53:56,658 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 2e3bfbeba47a4868ba61853e906e0f51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680490432220 2023-04-03 02:53:56,659 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 4319afea2ee346c0a0d8ad34d549101f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680490432851 2023-04-03 02:53:56,659 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6fa17bcf9e114c4d849f8b25ae9d7e56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680490433508 2023-04-03 02:53:56,660 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting b9291a29d2b647679f89771923cb0f5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680490434147 2023-04-03 02:53:56,661 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 91bfee7fd70f4a3b8a39d0bc9c00bc83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680490435283 2023-04-03 02:53:56,679 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#C#compaction#68 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:56,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490496814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490496814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490496815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:56,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:56,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490496815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,090 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/037c79c2fad045d7a75c4a8eb503b2d4 2023-04-03 02:53:57,101 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/3437d51aecad42dda50078e8dc47c1c0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/3437d51aecad42dda50078e8dc47c1c0 2023-04-03 02:53:57,106 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/f12cdb31fb504ce0a0be3f54d7c949f6 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/f12cdb31fb504ce0a0be3f54d7c949f6 2023-04-03 02:53:57,110 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/3437d51aecad42dda50078e8dc47c1c0, entries=150, sequenceid=483, filesize=11.9 K 2023-04-03 02:53:57,113 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/fc0c7bb99ae240b2987d9b13f3ca0b6b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/fc0c7bb99ae240b2987d9b13f3ca0b6b 2023-04-03 02:53:57,115 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/C of 459f839097be2d0597a0a35d5312fdc4 into f12cdb31fb504ce0a0be3f54d7c949f6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:57,116 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:57,116 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/C, priority=9, startTime=1680490436182; duration=0sec 2023-04-03 02:53:57,116 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:57,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490497117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490497118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,122 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/fc0c7bb99ae240b2987d9b13f3ca0b6b, entries=150, sequenceid=483, filesize=11.9 K 2023-04-03 02:53:57,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490497119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,124 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/037c79c2fad045d7a75c4a8eb503b2d4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/037c79c2fad045d7a75c4a8eb503b2d4 2023-04-03 02:53:57,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490497123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,131 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/037c79c2fad045d7a75c4a8eb503b2d4, entries=150, sequenceid=483, filesize=11.9 K 2023-04-03 02:53:57,133 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~401.34 KB/410976, currentSize=46.96 KB/48090 for 459f839097be2d0597a0a35d5312fdc4 in 939ms, sequenceid=483, compaction requested=false 2023-04-03 02:53:57,133 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:57,133 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:57,139 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:57,139 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:57,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:57,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@245de3ed[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:57,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:57,144 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:57,145 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:57,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:57,145 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,145 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:57,147 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:57,152 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:57,152 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:57,152 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:57,152 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:57,152 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:57,154 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:57,154 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:57,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:57,154 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:57,155 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:57,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:57,157 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,161 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:57,161 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:57,161 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,161 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:57,161 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:57,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:57,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:57,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:57,161 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:57,162 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,162 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:57,162 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:57,162 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:57,162 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:57,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:57,162 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:57,262 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:57,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:57,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:57,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:57,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:57,282 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:57,282 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:57,282 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:57,282 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:57,282 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:57,282 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:57,284 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:57,284 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:57,284 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,285 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:57,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:57,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:57,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:57,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:57,288 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:57,288 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:57,289 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:57,289 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:57,289 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:53:57,290 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:57,290 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:57,290 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:57,290 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:57,290 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:57,290 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:57,523 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:57,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:57,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490437586, max timestamp=1680490437616Num uniques 859; 2023-04-03 02:53:57,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:53:57,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490437586, max timestamp=1680490437616Num uniques 859; 2023-04-03 02:53:57,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490437586, max timestamp=1680490437616Num uniques 859; 2023-04-03 02:53:57,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:53:57,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:53:57,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490497665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490497666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490497676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490497676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490497676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,768 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/6131c29d2aae4e7ba011e5a1a637aef2 2023-04-03 02:53:57,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490497778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490497778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490497782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490497782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490497782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,790 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/7f700caa81ae4df1b085e9b354306645 2023-04-03 02:53:57,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490497980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490497980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490497985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490497986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:57,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:57,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490497986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,207 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/9294ba4c3e2b4183b9d627a37768b857 2023-04-03 02:53:58,213 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/6131c29d2aae4e7ba011e5a1a637aef2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6131c29d2aae4e7ba011e5a1a637aef2 2023-04-03 02:53:58,222 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6131c29d2aae4e7ba011e5a1a637aef2, entries=150, sequenceid=496, filesize=11.9 K 2023-04-03 02:53:58,223 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/7f700caa81ae4df1b085e9b354306645 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7f700caa81ae4df1b085e9b354306645 2023-04-03 02:53:58,228 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7f700caa81ae4df1b085e9b354306645, entries=150, sequenceid=496, filesize=11.9 K 2023-04-03 02:53:58,229 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/9294ba4c3e2b4183b9d627a37768b857 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/9294ba4c3e2b4183b9d627a37768b857 2023-04-03 02:53:58,236 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/9294ba4c3e2b4183b9d627a37768b857, entries=150, sequenceid=496, filesize=11.9 K 2023-04-03 02:53:58,237 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 459f839097be2d0597a0a35d5312fdc4 in 948ms, sequenceid=496, compaction requested=true 2023-04-03 02:53:58,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:58,238 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:58,238 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:58,238 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:58,238 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,238 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:58,238 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:58,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:58,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60dd41e0[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:58,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:58,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:58,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:58,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:58,244 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,244 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:58,246 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:58,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:58,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:58,250 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:58,250 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:58,250 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:58,250 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:58,250 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:58,252 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,252 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,252 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:58,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,257 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:58,257 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:58,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:58,257 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,257 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,257 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,257 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,258 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,258 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,258 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,259 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:58,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:58,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:58,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:58,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:58,259 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:58,259 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:58,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:58,287 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-03 02:53:58,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:58,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 02:53:58,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:58,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 02:53:58,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:58,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 02:53:58,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490498291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490498297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490498299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490498298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490498299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/9c787892aa844ede82dbd198ba1d0b67 2023-04-03 02:53:58,359 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:58,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:58,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:58,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:58,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:58,364 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:58,364 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:58,364 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:58,364 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:58,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,366 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:58,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:58,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:58,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:58,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:58,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:58,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:58,369 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:58,369 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:58,369 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:58,369 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:58,369 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,369 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:58,369 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:58,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,371 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:58,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38457657[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:58,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:58,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,374 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:58,381 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,381 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:58,384 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:58,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:58,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:58,387 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:58,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:58,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:58,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:58,387 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:58,389 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,389 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,390 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:58,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,392 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,394 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:58,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:58,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:58,398 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:58,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:58,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:58,398 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:58,398 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:58,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:58,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490498401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490498405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490498406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490498406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490498407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,498 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:58,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:58,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:58,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:58,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:58,502 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:58,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:58,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:58,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:58,503 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,503 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:58,504 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:58,504 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,504 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:58,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:58,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:58,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:58,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:58,506 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:58,506 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:58,506 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:58,506 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:58,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:58,509 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,509 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,509 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,509 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:58,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa522a3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:58,511 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:58,511 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,514 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,515 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:58,515 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:58,515 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:58,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:58,516 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:58,516 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,516 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,516 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:58,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:58,526 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:58,526 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:58,526 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:58,526 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:58,526 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:58,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,532 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,532 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,534 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,537 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,537 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,537 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,537 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:58,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:58,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:58,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:58,537 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,537 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,537 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,537 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:58,537 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:58,537 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:58,538 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:58,537 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:58,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:58,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490498605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490498609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490498610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490498610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490498611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,638 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:58,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:58,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:58,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:58,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:58,641 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:58,641 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:58,641 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:58,641 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:58,641 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,641 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,643 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:58,643 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:58,643 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,643 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:58,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:58,644 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:58,644 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:58,644 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:58,644 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:58,644 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:58,644 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:58,644 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:58,645 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:58,645 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:58,645 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:58,645 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:58,645 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,645 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:58,645 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:58,646 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:58,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42cf231[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:58,648 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:58,648 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,650 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,650 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:58,650 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:58,650 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,650 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:58,650 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:58,651 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,652 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,652 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:58,652 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:58,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:58,655 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:58,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:58,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:58,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:58,655 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:58,657 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,657 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:58,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,657 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:58,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:58,659 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,661 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,663 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:58,663 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,663 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,663 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,663 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,663 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,663 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:58,663 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,663 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:58,663 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:58,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:58,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:58,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:58,664 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:58,664 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:58,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:58,731 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/33fcd00506d542e2bd82a648e47ea2ee 2023-04-03 02:53:58,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/3c38d2a48f34445ca209c9303227a4d9 2023-04-03 02:53:58,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/9c787892aa844ede82dbd198ba1d0b67 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/9c787892aa844ede82dbd198ba1d0b67 2023-04-03 02:53:58,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/9c787892aa844ede82dbd198ba1d0b67, entries=200, sequenceid=534, filesize=14.3 K 2023-04-03 02:53:58,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/33fcd00506d542e2bd82a648e47ea2ee as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/33fcd00506d542e2bd82a648e47ea2ee 2023-04-03 02:53:58,764 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:58,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:58,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/33fcd00506d542e2bd82a648e47ea2ee, entries=150, sequenceid=534, filesize=11.9 K 2023-04-03 02:53:58,767 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/3c38d2a48f34445ca209c9303227a4d9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3c38d2a48f34445ca209c9303227a4d9 2023-04-03 02:53:58,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:58,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:58,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:58,772 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:58,772 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:58,772 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:58,772 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:58,773 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,773 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3c38d2a48f34445ca209c9303227a4d9, entries=150, sequenceid=534, filesize=11.9 K 2023-04-03 02:53:58,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,775 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:58,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:58,776 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for 459f839097be2d0597a0a35d5312fdc4 in 488ms, sequenceid=534, compaction requested=true 2023-04-03 02:53:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:58,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:58,776 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,776 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:58,776 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:58,776 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:53:58,776 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-03 02:53:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,776 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:53:58,776 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:53:58,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:58,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:58,778 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:53:58,778 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/A is initiating minor compaction (all files) 2023-04-03 02:53:58,778 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:53:58,778 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/A in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:58,778 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/C is initiating minor compaction (all files) 2023-04-03 02:53:58,778 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/e6596c3216174329aacd0121b9cbb2ab, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/3437d51aecad42dda50078e8dc47c1c0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6131c29d2aae4e7ba011e5a1a637aef2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/9c787892aa844ede82dbd198ba1d0b67] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=50.7 K 2023-04-03 02:53:58,778 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/C in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:58,778 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/f12cdb31fb504ce0a0be3f54d7c949f6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/037c79c2fad045d7a75c4a8eb503b2d4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/9294ba4c3e2b4183b9d627a37768b857, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3c38d2a48f34445ca209c9303227a4d9] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=48.3 K 2023-04-03 02:53:58,779 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e6596c3216174329aacd0121b9cbb2ab, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680490435283 2023-04-03 02:53:58,779 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f12cdb31fb504ce0a0be3f54d7c949f6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680490435283 2023-04-03 02:53:58,779 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 3437d51aecad42dda50078e8dc47c1c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1680490435346 2023-04-03 02:53:58,779 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 037c79c2fad045d7a75c4a8eb503b2d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1680490435346 2023-04-03 02:53:58,780 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6131c29d2aae4e7ba011e5a1a637aef2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680490436501 2023-04-03 02:53:58,780 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 9294ba4c3e2b4183b9d627a37768b857, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680490436501 2023-04-03 02:53:58,780 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 9c787892aa844ede82dbd198ba1d0b67, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680490437674 2023-04-03 02:53:58,780 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 3c38d2a48f34445ca209c9303227a4d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680490437674 2023-04-03 02:53:58,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:58,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:58,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:58,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:58,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:58,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:58,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:58,782 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:58,782 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:58,783 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-03 02:53:58,784 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:58,784 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:58,784 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:58,784 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:58,784 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:58,784 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:53:58,815 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#A#compaction#75 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:58,815 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#C#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:58,863 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/7567630bbb1d43edafd5a2202de9ec4c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/7567630bbb1d43edafd5a2202de9ec4c 2023-04-03 02:53:58,873 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/C of 459f839097be2d0597a0a35d5312fdc4 into 7567630bbb1d43edafd5a2202de9ec4c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:58,873 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:58,873 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/C, priority=12, startTime=1680490438776; duration=0sec 2023-04-03 02:53:58,873 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:53:58,873 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:53:58,876 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:53:58,876 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/B is initiating minor compaction (all files) 2023-04-03 02:53:58,876 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/B in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:58,876 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/802683618e5445eb9bcb972c2568661e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/fc0c7bb99ae240b2987d9b13f3ca0b6b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7f700caa81ae4df1b085e9b354306645, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/33fcd00506d542e2bd82a648e47ea2ee] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=48.3 K 2023-04-03 02:53:58,877 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 802683618e5445eb9bcb972c2568661e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1680490435283 2023-04-03 02:53:58,878 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting fc0c7bb99ae240b2987d9b13f3ca0b6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1680490435346 2023-04-03 02:53:58,878 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7f700caa81ae4df1b085e9b354306645, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680490436501 2023-04-03 02:53:58,879 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 33fcd00506d542e2bd82a648e47ea2ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680490437674 2023-04-03 02:53:58,892 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/ff66486a9909429da7b4018b8d149148 2023-04-03 02:53:58,901 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#B#compaction#78 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:53:58,914 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:58,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:58,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490438929, max timestamp=1680490438935Num uniques 859; 2023-04-03 02:53:58,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:53:58,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490438929, max timestamp=1680490438935Num uniques 859; 2023-04-03 02:53:58,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:53:58,944 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/0789239519864c75be770feba2bbee83 2023-04-03 02:53:58,945 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/32e08ecc422f4aee8945d1084df12aa9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/32e08ecc422f4aee8945d1084df12aa9 2023-04-03 02:53:58,957 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/B of 459f839097be2d0597a0a35d5312fdc4 into 32e08ecc422f4aee8945d1084df12aa9(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:58,957 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:58,958 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/B, priority=12, startTime=1680490438776; duration=0sec 2023-04-03 02:53:58,958 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:58,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490498958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490498959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490498959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490498960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:58,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:58,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490498967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490499068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490499068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490499068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490499069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490499069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490499270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490499271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490499272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490499273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490499273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,309 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/bc39edcad2cd4ab9be1f513bcd4fe523 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/bc39edcad2cd4ab9be1f513bcd4fe523 2023-04-03 02:53:59,315 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/A of 459f839097be2d0597a0a35d5312fdc4 into bc39edcad2cd4ab9be1f513bcd4fe523(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:53:59,315 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:59,315 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/A, priority=12, startTime=1680490438776; duration=0sec 2023-04-03 02:53:59,315 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:53:59,363 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/6424c06368c0416d834ae08f16eeb92e 2023-04-03 02:53:59,369 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/ff66486a9909429da7b4018b8d149148 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ff66486a9909429da7b4018b8d149148 2023-04-03 02:53:59,375 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ff66486a9909429da7b4018b8d149148, entries=150, sequenceid=542, filesize=11.9 K 2023-04-03 02:53:59,376 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/0789239519864c75be770feba2bbee83 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/0789239519864c75be770feba2bbee83 2023-04-03 02:53:59,381 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/0789239519864c75be770feba2bbee83, entries=150, sequenceid=542, filesize=11.9 K 2023-04-03 02:53:59,382 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/6424c06368c0416d834ae08f16eeb92e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6424c06368c0416d834ae08f16eeb92e 2023-04-03 02:53:59,386 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6424c06368c0416d834ae08f16eeb92e, entries=150, sequenceid=542, filesize=11.9 K 2023-04-03 02:53:59,387 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=145.36 KB/148850 for 459f839097be2d0597a0a35d5312fdc4 in 605ms, sequenceid=542, compaction requested=false 2023-04-03 02:53:59,387 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:53:59,387 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:59,441 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:59,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:59,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:59,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e746115[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:59,444 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:59,444 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,487 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,488 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:59,488 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:59,488 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:59,488 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,488 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:59,541 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:59,544 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:59,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:59,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:59,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:59,544 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:59,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:53:59,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=152.07 KB heapSize=443.08 KB 2023-04-03 02:53:59,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:53:59,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:59,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:53:59,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:59,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:53:59,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:53:59,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490499582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490499582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490499582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490499583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490499583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,587 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,588 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,588 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:59,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:59,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,594 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,595 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:59,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,642 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:59,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:59,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:59,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:59,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:59,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:59,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:59,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:59,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,642 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:59,642 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:59,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:59,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490499686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490499687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,742 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:53:59,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:53:59,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:53:59,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:53:59,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:53:59,746 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:53:59,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:59,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:53:59,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:53:59,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,747 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:59,803 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:59,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:59,803 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,803 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:53:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:59,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:53:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:59,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:53:59,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:53:59,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:53:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:53:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:53:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:53:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:53:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:53:59,805 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:59,805 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:53:59,805 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:53:59,805 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:53:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:53:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:53:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:53:59,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:53:59,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,857 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:59,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:53:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:59,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:53:59,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d904942[Count = 0] remaining members to acquire global barrier 2023-04-03 02:53:59,859 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:53:59,859 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490499889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:53:59,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490499889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,910 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:53:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:53:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:53:59,911 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,911 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:53:59,986 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:53:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:59,987 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:59,987 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:53:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:53:59,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:53:59,990 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:53:59,990 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:53:59,990 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:53:59,990 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:53:59,990 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:53:59,993 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,993 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,993 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:53:59,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:53:59,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,995 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:53:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:53:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:59,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:53:59,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:53:59,998 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:53:59,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:53:59,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:53:59,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:53:59,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:53:59,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:53:59,999 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:53:59,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:53:59,999 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:53:59,999 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:53:59,999 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:53:59,999 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:53:59,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:53:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:53:59,999 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/0cf3f43bceb242e0bb4411c0bc745c82 2023-04-03 02:54:00,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:00,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490500085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:00,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490500088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:00,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490500089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,099 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:00,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:00,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:00,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:00,102 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:00,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:00,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:00,103 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,103 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,104 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,104 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,105 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,105 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:00,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:00,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:00,106 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,106 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:00,106 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:00,106 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:00,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:00,108 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,108 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,109 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,109 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:00,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bc6d069[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:00,110 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:00,110 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,112 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:00,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:00,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:00,113 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,113 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:00,114 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,114 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:00,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:00,117 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:00,117 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:00,117 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:00,117 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:00,117 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:00,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,120 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:00,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:00,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,127 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:00,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,127 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:00,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:00,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490500193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:00,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490500193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,227 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:00,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:00,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:00,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:00,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:00,231 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:00,231 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:00,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:00,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,234 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,235 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,235 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,235 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:00,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:00,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:00,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:00,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:00,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:00,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:00,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:00,241 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,241 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:00,241 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:00,241 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:00,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:00,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:00,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,245 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,252 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,252 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:00,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1818185a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:00,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:00,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,255 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:00,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:00,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:00,258 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:00,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:00,261 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:00,261 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:00,261 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:00,261 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:00,261 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:00,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,267 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,269 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,269 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,269 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,269 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:00,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:00,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,269 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:00,269 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:00,269 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,269 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,269 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,270 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,270 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,369 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:00,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:00,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:00,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:00,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:00,373 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:00,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:00,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:00,374 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,374 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,375 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,375 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,375 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:00,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:00,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:00,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:00,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:00,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:00,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:00,377 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,377 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:00,377 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:00,377 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:00,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:00,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:00,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,379 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,379 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:00,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3549ae85[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:00,381 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:00,381 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,383 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,383 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:00,383 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:00,383 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:00,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:00,384 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:00,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:00,387 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:00,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:00,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:00,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:00,387 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:00,389 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,389 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,389 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,391 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:00,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:00,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:00,398 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:00,398 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:00,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,398 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,398 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/41d4d4b60c094f7aa911b184d37bfa6f 2023-04-03 02:54:00,498 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:00,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:00,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:00,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:00,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:00,501 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:00,501 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:00,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:00,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,503 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,503 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,504 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,504 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:00,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:00,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:00,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:00,505 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,506 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:00,506 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:00,506 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:00,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:00,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:00,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:00,507 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,507 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,508 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,508 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:00,509 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:00,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a8c2b37[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:00,509 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,510 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,511 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:00,511 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,511 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:00,511 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:00,511 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:00,518 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:00,518 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,518 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:00,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:00,521 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:00,521 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:00,521 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:00,521 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:00,521 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:00,522 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,523 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,524 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,524 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,524 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,527 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,527 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,527 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,527 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,527 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:00,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:00,527 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:00,527 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,527 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:00,527 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,527 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:00,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,628 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:00,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:00,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:00,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:00,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:00,631 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:00,631 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,631 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:00,631 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:00,632 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,632 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,633 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,633 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,633 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,633 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:00,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:00,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:00,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:00,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:00,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:00,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:00,635 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,635 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:00,635 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:00,635 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:00,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:00,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:00,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:00,637 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,638 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:00,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6784f397[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:00,640 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:00,640 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:00,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:00,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:00,641 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,641 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:00,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,643 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,643 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,643 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:00,645 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:00,646 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:00,646 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:00,646 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:00,646 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:00,647 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,647 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,647 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,648 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,650 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,653 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,653 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:00,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:00,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:00,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,653 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,653 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,654 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,654 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,654 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,654 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,654 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,655 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:00,655 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:00,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490500696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490500697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,755 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:00,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:00,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:00,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:00,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:00,759 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:00,759 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,759 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:00,759 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:00,760 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,760 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,761 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,761 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,761 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,762 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:00,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:00,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:00,763 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:00,763 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,763 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:00,763 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:00,763 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:00,763 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,763 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:00,763 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:00,763 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,763 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:00,763 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:00,763 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,764 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:00,764 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:00,765 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,765 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,765 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,765 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f86edf5[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:00,767 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:00,767 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,769 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:00,769 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,769 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:00,769 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:00,771 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:00,771 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,771 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,771 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:00,773 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:00,773 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:00,773 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:00,774 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:00,774 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:00,775 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,775 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,775 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:00,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,776 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:00,777 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:00,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:00,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:00,783 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:00,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:00,783 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:00,783 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:00,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:00,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:00,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=71.56 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/71db2c88178242f7924bd6bd1a7f8c58 2023-04-03 02:54:00,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/0cf3f43bceb242e0bb4411c0bc745c82 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/0cf3f43bceb242e0bb4411c0bc745c82 2023-04-03 02:54:00,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/0cf3f43bceb242e0bb4411c0bc745c82, entries=200, sequenceid=580, filesize=14.3 K 2023-04-03 02:54:00,849 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/41d4d4b60c094f7aa911b184d37bfa6f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/41d4d4b60c094f7aa911b184d37bfa6f 2023-04-03 02:54:00,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/41d4d4b60c094f7aa911b184d37bfa6f, entries=150, sequenceid=580, filesize=11.9 K 2023-04-03 02:54:00,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/71db2c88178242f7924bd6bd1a7f8c58 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/71db2c88178242f7924bd6bd1a7f8c58 2023-04-03 02:54:00,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/71db2c88178242f7924bd6bd1a7f8c58, entries=150, sequenceid=580, filesize=11.9 K 2023-04-03 02:54:00,869 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~152.07 KB/155720, heapSize ~443.03 KB/453664, currentSize=26.84 KB/27480 for 459f839097be2d0597a0a35d5312fdc4 in 1294ms, sequenceid=580, compaction requested=true 2023-04-03 02:54:00,869 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:00,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:00,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:00,869 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:00,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:54:00,870 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:00,871 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:00,871 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39850 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:00,871 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/B is initiating minor compaction (all files) 2023-04-03 02:54:00,871 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/A is initiating minor compaction (all files) 2023-04-03 02:54:00,872 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/B in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,872 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/A in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,872 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/32e08ecc422f4aee8945d1084df12aa9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/0789239519864c75be770feba2bbee83, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/41d4d4b60c094f7aa911b184d37bfa6f] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=36.5 K 2023-04-03 02:54:00,872 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/bc39edcad2cd4ab9be1f513bcd4fe523, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ff66486a9909429da7b4018b8d149148, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/0cf3f43bceb242e0bb4411c0bc745c82] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=38.9 K 2023-04-03 02:54:00,872 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 32e08ecc422f4aee8945d1084df12aa9, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680490437674 2023-04-03 02:54:00,872 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting bc39edcad2cd4ab9be1f513bcd4fe523, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680490437674 2023-04-03 02:54:00,873 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 0789239519864c75be770feba2bbee83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1680490438294 2023-04-03 02:54:00,873 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting ff66486a9909429da7b4018b8d149148, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1680490438294 2023-04-03 02:54:00,873 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 41d4d4b60c094f7aa911b184d37bfa6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680490438958 2023-04-03 02:54:00,873 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 0cf3f43bceb242e0bb4411c0bc745c82, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680490438958 2023-04-03 02:54:00,884 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:00,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:00,885 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#B#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:00,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:00,887 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#A#compaction#85 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:00,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:00,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:00,893 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:00,893 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,893 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:00,893 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:00,894 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,894 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,895 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:00,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:00,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:00,895 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:00,897 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:00,897 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:00,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:00,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:00,897 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:00,897 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:00,897 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:00,900 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:00,900 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:00,900 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,900 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:00,900 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 02:54:00,900 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:54:00,901 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:00,901 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:54:00,901 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:00,901 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:54:00,901 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:00,910 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/a9c24060c5b5491eb199319b6946995f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a9c24060c5b5491eb199319b6946995f 2023-04-03 02:54:00,915 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/0771ed9e2d144e53aeb8621550edc906 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/0771ed9e2d144e53aeb8621550edc906 2023-04-03 02:54:00,928 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/A of 459f839097be2d0597a0a35d5312fdc4 into a9c24060c5b5491eb199319b6946995f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:00,929 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:00,929 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/A, priority=13, startTime=1680490440869; duration=0sec 2023-04-03 02:54:00,929 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:00,929 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:00,931 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/B of 459f839097be2d0597a0a35d5312fdc4 into 0771ed9e2d144e53aeb8621550edc906(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:00,931 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:00,931 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/B, priority=13, startTime=1680490440869; duration=0sec 2023-04-03 02:54:00,931 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:00,931 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:00,931 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/C is initiating minor compaction (all files) 2023-04-03 02:54:00,931 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/C in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:00,931 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/7567630bbb1d43edafd5a2202de9ec4c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6424c06368c0416d834ae08f16eeb92e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/71db2c88178242f7924bd6bd1a7f8c58] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=36.5 K 2023-04-03 02:54:00,932 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 7567630bbb1d43edafd5a2202de9ec4c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1680490437674 2023-04-03 02:54:00,932 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6424c06368c0416d834ae08f16eeb92e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1680490438294 2023-04-03 02:54:00,932 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 71db2c88178242f7924bd6bd1a7f8c58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680490438958 2023-04-03 02:54:00,941 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#C#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:01,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:54:01,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:01,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490441112, max timestamp=1680490441116Num uniques 859; 2023-04-03 02:54:01,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:54:01,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490441112, max timestamp=1680490441116Num uniques 859; 2023-04-03 02:54:01,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:54:01,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490441112, max timestamp=1680490441116Num uniques 859; 2023-04-03 02:54:01,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:54:01,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1680490441125, max timestamp=1680490441141Num uniques 1018; 2023-04-03 02:54:01,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=19138, suffix heap size=133840,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=418 2023-04-03 02:54:01,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490441125, max timestamp=1680490441141Num uniques 1018; 2023-04-03 02:54:01,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-03 02:54:01,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490441125, max timestamp=1680490441141Num uniques 1018; 2023-04-03 02:54:01,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-03 02:54:01,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680490441159, max timestamp=1680490441165Num uniques 1277; 2023-04-03 02:54:01,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680490441159, max timestamp=1680490441165Num uniques 1127; 2023-04-03 02:54:01,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680490441159, max timestamp=1680490441165Num uniques 1127; 2023-04-03 02:54:01,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-03 02:54:01,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=58482, new segment data size=8102, suffix heap size=160776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1277, new segment cells count=177 2023-04-03 02:54:01,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-03 02:54:01,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1680490441173, max timestamp=1680490441189Num uniques 1036; 2023-04-03 02:54:01,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=19966, suffix heap size=135712,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=436 2023-04-03 02:54:01,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1680490441173, max timestamp=1680490441189Num uniques 1036; 2023-04-03 02:54:01,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=19966, suffix heap size=135712,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=436 2023-04-03 02:54:01,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1680490441173, max timestamp=1680490441189Num uniques 1036; 2023-04-03 02:54:01,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=19966, suffix heap size=135712,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=436 2023-04-03 02:54:01,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490501209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490501210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490501210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490501313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490501313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490501313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,342 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=587 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/746672cc61d64bdc9ba77474b953fa8b 2023-04-03 02:54:01,372 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/a9b33cd7237b45fd8b3d5435984cf99b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a9b33cd7237b45fd8b3d5435984cf99b 2023-04-03 02:54:01,377 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/C of 459f839097be2d0597a0a35d5312fdc4 into a9b33cd7237b45fd8b3d5435984cf99b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:01,377 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:01,378 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/C, priority=13, startTime=1680490440869; duration=0sec 2023-04-03 02:54:01,378 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:01,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490501517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490501517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490501517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490501703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490501707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,756 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=587 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/8487fe51e9aa44ad8af6931df091ab88 2023-04-03 02:54:01,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490501820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490501820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:01,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:01,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490501821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,170 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=587 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/5c4d857408844a6ebd6f637a9846611d 2023-04-03 02:54:02,176 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/746672cc61d64bdc9ba77474b953fa8b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/746672cc61d64bdc9ba77474b953fa8b 2023-04-03 02:54:02,182 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/746672cc61d64bdc9ba77474b953fa8b, entries=100, sequenceid=587, filesize=9.5 K 2023-04-03 02:54:02,185 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/8487fe51e9aa44ad8af6931df091ab88 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/8487fe51e9aa44ad8af6931df091ab88 2023-04-03 02:54:02,202 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/8487fe51e9aa44ad8af6931df091ab88, entries=100, sequenceid=587, filesize=9.5 K 2023-04-03 02:54:02,203 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/5c4d857408844a6ebd6f637a9846611d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/5c4d857408844a6ebd6f637a9846611d 2023-04-03 02:54:02,214 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/5c4d857408844a6ebd6f637a9846611d, entries=100, sequenceid=587, filesize=9.5 K 2023-04-03 02:54:02,218 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=167.72 KB/171750 for 459f839097be2d0597a0a35d5312fdc4 in 1318ms, sequenceid=587, compaction requested=false 2023-04-03 02:54:02,218 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:02,218 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:02,218 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:02,218 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:02,218 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,218 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:02,218 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:02,220 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,221 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,221 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,221 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:02,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c34106[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:02,223 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:02,223 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,224 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,224 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:02,224 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:02,224 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:02,224 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,225 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:02,226 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:02,226 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,226 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:02,226 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:02,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:02,237 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:02,237 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:02,237 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:02,237 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:02,237 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:02,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,240 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:02,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:02,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:02,247 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:02,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:02,247 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:02,247 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:02,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,248 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,248 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:02,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:54:02,325 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=174.43 KB heapSize=479.25 KB 2023-04-03 02:54:02,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680490441205, max timestamp=1680490442323Num uniques 1295; 2023-04-03 02:54:02,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=59310, new segment data size=8930, suffix heap size=162648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1295, new segment cells count=195 2023-04-03 02:54:02,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680490441205, max timestamp=1680490442323Num uniques 1295; 2023-04-03 02:54:02,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680490441205, max timestamp=1680490442323Num uniques 1295; 2023-04-03 02:54:02,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=59310, new segment data size=8930, suffix heap size=162648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1295, new segment cells count=195 2023-04-03 02:54:02,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=59310, new segment data size=8930, suffix heap size=162648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1295, new segment cells count=195 2023-04-03 02:54:02,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:54:02,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:02,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:54:02,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:02,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:54:02,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:02,347 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:02,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:02,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490442342, max timestamp=1680490442349Num uniques 859; 2023-04-03 02:54:02,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:54:02,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490442342, max timestamp=1680490442349Num uniques 859; 2023-04-03 02:54:02,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:54:02,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490442342, max timestamp=1680490442349Num uniques 859; 2023-04-03 02:54:02,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:54:02,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:02,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:02,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:02,355 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:02,355 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:02,355 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:02,355 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:02,355 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,355 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=680 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/ac7deb6c9adb4434a10fa50e65644aac 2023-04-03 02:54:02,357 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:02,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:02,357 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,358 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,358 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:02,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:02,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:02,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:02,360 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:02,360 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:02,360 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:02,360 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,361 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:02,361 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:02,365 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,365 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,365 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,366 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:02,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c5d1f5a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:02,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:02,369 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,370 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,370 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:02,370 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:02,370 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:02,370 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,370 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:02,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,376 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:02,376 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,376 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:02,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:02,378 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:02,378 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:02,378 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:02,378 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:02,378 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:02,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:02,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490502375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490442368, max timestamp=1680490442378Num uniques 1118; 2023-04-03 02:54:02,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-03 02:54:02,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680490442363, max timestamp=1680490442378Num uniques 1118; 2023-04-03 02:54:02,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-03 02:54:02,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490442368, max timestamp=1680490442378Num uniques 1118; 2023-04-03 02:54:02,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-03 02:54:02,381 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,381 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,381 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:02,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:02,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:02,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:02,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:02,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,388 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,388 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:02,388 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:02,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680490442387, max timestamp=1680490442405Num uniques 1177; 2023-04-03 02:54:02,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680490442387, max timestamp=1680490442405Num uniques 1027; 2023-04-03 02:54:02,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680490442387, max timestamp=1680490442405Num uniques 1027; 2023-04-03 02:54:02,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-03 02:54:02,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-03 02:54:02,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-03 02:54:02,437 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680490442422, max timestamp=1680490442435Num uniques 1136; 2023-04-03 02:54:02,437 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-04-03 02:54:02,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680490442422, max timestamp=1680490442435Num uniques 1136; 2023-04-03 02:54:02,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680490442422, max timestamp=1680490442435Num uniques 1136; 2023-04-03 02:54:02,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-04-03 02:54:02,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-04-03 02:54:02,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680490442461, max timestamp=1680490442476Num uniques 1195; 2023-04-03 02:54:02,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-03 02:54:02,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680490442461, max timestamp=1680490442476Num uniques 1195; 2023-04-03 02:54:02,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-03 02:54:02,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680490442461, max timestamp=1680490442476Num uniques 1195; 2023-04-03 02:54:02,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-03 02:54:02,488 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:02,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:02,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:02,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:02,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:02,491 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:02,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:02,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:02,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:02,492 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,492 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,493 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:02,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:02,494 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,494 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:02,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:02,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:02,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:02,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:02,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:02,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:02,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:02,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:02,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:02,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:02,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:02,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:02,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:02,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:02,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:02,498 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,498 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,498 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,498 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:02,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:02,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c055d0d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:02,500 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:02,500 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,501 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,501 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:02,501 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:02,501 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:02,502 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:02,502 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,502 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:02,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680490442491, max timestamp=1680490442502Num uniques 1054; 2023-04-03 02:54:02,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-03 02:54:02,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680490442491, max timestamp=1680490442502Num uniques 1054; 2023-04-03 02:54:02,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680490442491, max timestamp=1680490442502Num uniques 1054; 2023-04-03 02:54:02,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-03 02:54:02,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-03 02:54:02,505 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:02,505 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:02,505 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:02,505 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:02,505 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:02,505 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:02,505 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:02,507 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,507 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,507 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,508 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,509 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:02,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:02,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:02,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,515 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:02,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:02,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:02,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,515 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:02,515 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:02,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680490442520, max timestamp=1680490442552Num uniques 1113; 2023-04-03 02:54:02,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680490442520, max timestamp=1680490442552Num uniques 1113; 2023-04-03 02:54:02,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:02,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680490442520, max timestamp=1680490442552Num uniques 1113; 2023-04-03 02:54:02,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-03 02:54:02,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-03 02:54:02,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-03 02:54:02,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490502552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680490442572, max timestamp=1680490442577Num uniques 1022; 2023-04-03 02:54:02,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-03 02:54:02,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680490442572, max timestamp=1680490442577Num uniques 1072; 2023-04-03 02:54:02,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680490442572, max timestamp=1680490442577Num uniques 1022; 2023-04-03 02:54:02,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-03 02:54:02,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-03 02:54:02,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680490442589, max timestamp=1680490442601Num uniques 1031; 2023-04-03 02:54:02,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680490442589, max timestamp=1680490442601Num uniques 1031; 2023-04-03 02:54:02,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680490442589, max timestamp=1680490442601Num uniques 1031; 2023-04-03 02:54:02,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=15156, suffix heap size=135192,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=331 2023-04-03 02:54:02,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=15156, suffix heap size=135192,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=331 2023-04-03 02:54:02,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=15156, suffix heap size=135192,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=331 2023-04-03 02:54:02,616 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:02,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:02,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:02,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:02,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:02,619 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:02,619 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:02,619 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:02,619 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:02,619 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,619 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,623 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:02,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:02,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:02,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:02,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:02,625 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:02,625 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:02,625 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:02,625 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:02,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:02,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,627 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,627 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,627 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:02,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4464e83d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:02,629 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:02,629 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680490442622, max timestamp=1680490442630Num uniques 1190; 2023-04-03 02:54:02,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=10990, suffix heap size=151728,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=240 2023-04-03 02:54:02,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680490442622, max timestamp=1680490442630Num uniques 1190; 2023-04-03 02:54:02,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=10990, suffix heap size=151728,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=240 2023-04-03 02:54:02,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680490442622, max timestamp=1680490442630Num uniques 1190; 2023-04-03 02:54:02,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=10990, suffix heap size=151728,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=240 2023-04-03 02:54:02,633 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,633 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:02,633 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:02,633 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:02,633 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,633 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:02,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:02,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,635 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,635 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:02,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:02,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:02,638 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:02,638 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:02,638 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:02,638 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:02,638 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:02,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,643 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,645 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,647 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,647 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:02,647 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:02,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:02,647 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:02,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:02,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:02,647 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,647 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,648 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,648 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,648 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:02,648 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:02,648 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,648 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:02,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490502660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680490442656, max timestamp=1680490442661Num uniques 1099; 2023-04-03 02:54:02,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680490442656, max timestamp=1680490442661Num uniques 1099; 2023-04-03 02:54:02,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-03 02:54:02,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-03 02:54:02,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680490442656, max timestamp=1680490442661Num uniques 1099; 2023-04-03 02:54:02,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-03 02:54:02,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680490442683, max timestamp=1680490442692Num uniques 1058; 2023-04-03 02:54:02,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-03 02:54:02,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680490442683, max timestamp=1680490442692Num uniques 1058; 2023-04-03 02:54:02,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-03 02:54:02,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680490442683, max timestamp=1680490442692Num uniques 1058; 2023-04-03 02:54:02,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-03 02:54:02,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1680490442704, max timestamp=1680490442723Num uniques 1067; 2023-04-03 02:54:02,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1680490442704, max timestamp=1680490442723Num uniques 1067; 2023-04-03 02:54:02,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=16802, suffix heap size=138936,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=367 2023-04-03 02:54:02,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=16802, suffix heap size=138936,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=367 2023-04-03 02:54:02,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680490442716, max timestamp=1680490442723Num uniques 1067; 2023-04-03 02:54:02,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-03 02:54:02,748 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:02,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:02,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:02,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:02,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:02,752 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:02,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:02,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:02,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:02,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,755 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:02,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:02,755 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:02,755 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,756 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:02,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:02,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:02,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:02,757 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:02,757 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:02,757 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:02,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:02,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490502764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,768 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:02,768 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:02,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=680 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/567cc6a8d18b4e1f8e919776e96b62f8 2023-04-03 02:54:02,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:02,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490502767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:02,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490502767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,772 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:02,772 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:02,772 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:02,772 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:02,774 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:02,774 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:02,774 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,774 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:02,774 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:02,778 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,778 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,779 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,779 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:02,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19725ac6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:02,781 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:02,781 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,782 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,782 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:02,782 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:02,782 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:02,786 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,786 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:02,786 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:02,786 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,786 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,787 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:02,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:02,790 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:02,790 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:02,790 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:02,790 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:02,790 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:02,793 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,793 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,793 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,799 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:02,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:02,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:02,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:02,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,799 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:02,799 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:02,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:02,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:02,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490502873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:02,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490502873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,900 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:02,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:02,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:02,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:02,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:02,904 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:02,904 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:02,904 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:02,904 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:02,904 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,904 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,906 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:02,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:02,906 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,906 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,906 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:02,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:02,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:02,908 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:02,908 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:02,908 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:02,908 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:02,908 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:02,908 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:02,908 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:02,908 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:02,909 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:02,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:02,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:02,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:02,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:02,911 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:02,918 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:02,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7db1217d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:02,918 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:02,922 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:02,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:02,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:02,924 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:02,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:02,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:02,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:02,927 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:02,927 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:02,927 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:02,927 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:02,927 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:02,929 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,929 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,929 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:02,929 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:02,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:02,931 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:02,933 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,933 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:02,933 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:02,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:02,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:02,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:02,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,933 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:02,934 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:02,934 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:02,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:02,933 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:02,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:02,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:02,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:02,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:02,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:02,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490502969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,034 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:03,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:03,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:03,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:03,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:03,038 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:03,038 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,038 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:03,038 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:03,039 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,039 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,040 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,041 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,041 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:03,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:03,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:03,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:03,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:03,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:03,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:03,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:03,055 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,055 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:03,055 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:03,055 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:03,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:03,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:03,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:03,057 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,057 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,057 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:03,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:03,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bbae6f3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:03,062 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:03,062 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,066 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,067 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:03,067 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:03,067 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:03,067 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,067 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,069 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:03,069 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:03,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:03,072 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:03,072 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:03,072 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:03,072 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:03,072 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:03,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,076 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490503079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490503080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,084 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,087 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:03,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:03,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,088 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,088 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,088 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,088 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:03,088 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,188 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:03,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:03,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:03,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:03,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:03,192 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:03,192 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,192 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:03,192 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:03,193 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,193 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,196 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,197 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,197 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:03,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:03,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:03,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:03,198 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:03,198 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,199 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:03,199 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:03,199 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,199 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:03,199 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:03,199 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,199 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:03,199 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:03,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,200 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,201 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,201 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:03,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fcc0b9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:03,202 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:03,202 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=680 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/0c56e9b687d84095bddb40b1c9c37224 2023-04-03 02:54:03,203 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,204 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:03,204 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:03,204 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:03,204 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,204 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:03,205 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:03,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/ac7deb6c9adb4434a10fa50e65644aac as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ac7deb6c9adb4434a10fa50e65644aac 2023-04-03 02:54:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:03,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:03,209 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:03,209 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:03,210 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:03,210 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:03,210 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:03,212 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,212 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,212 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,213 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ac7deb6c9adb4434a10fa50e65644aac, entries=200, sequenceid=680, filesize=14.3 K 2023-04-03 02:54:03,215 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/567cc6a8d18b4e1f8e919776e96b62f8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/567cc6a8d18b4e1f8e919776e96b62f8 2023-04-03 02:54:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,218 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,218 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,218 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:03,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:03,218 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,218 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,218 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,218 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,218 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,219 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:03,219 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:03,219 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,219 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:03,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/567cc6a8d18b4e1f8e919776e96b62f8, entries=150, sequenceid=680, filesize=11.9 K 2023-04-03 02:54:03,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/0c56e9b687d84095bddb40b1c9c37224 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/0c56e9b687d84095bddb40b1c9c37224 2023-04-03 02:54:03,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/0c56e9b687d84095bddb40b1c9c37224, entries=150, sequenceid=680, filesize=11.9 K 2023-04-03 02:54:03,228 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~82.73 KB/84720, currentSize=154.31 KB/158010 for 459f839097be2d0597a0a35d5312fdc4 in 903ms, sequenceid=680, compaction requested=true 2023-04-03 02:54:03,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:03,228 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:03,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:03,229 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:03,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:03,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:03,230 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37508 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:03,231 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/A is initiating minor compaction (all files) 2023-04-03 02:54:03,231 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/A in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,231 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a9c24060c5b5491eb199319b6946995f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/746672cc61d64bdc9ba77474b953fa8b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ac7deb6c9adb4434a10fa50e65644aac] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=36.6 K 2023-04-03 02:54:03,231 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a9c24060c5b5491eb199319b6946995f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680490438958 2023-04-03 02:54:03,235 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 746672cc61d64bdc9ba77474b953fa8b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1680490439579 2023-04-03 02:54:03,236 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:03,236 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/B is initiating minor compaction (all files) 2023-04-03 02:54:03,236 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/B in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,236 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/0771ed9e2d144e53aeb8621550edc906, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/8487fe51e9aa44ad8af6931df091ab88, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/567cc6a8d18b4e1f8e919776e96b62f8] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=34.2 K 2023-04-03 02:54:03,237 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting ac7deb6c9adb4434a10fa50e65644aac, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1680490441205 2023-04-03 02:54:03,238 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 0771ed9e2d144e53aeb8621550edc906, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680490438958 2023-04-03 02:54:03,240 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 8487fe51e9aa44ad8af6931df091ab88, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1680490439579 2023-04-03 02:54:03,241 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 567cc6a8d18b4e1f8e919776e96b62f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1680490441209 2023-04-03 02:54:03,256 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#B#compaction#93 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:03,257 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#A#compaction#94 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:03,284 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/44df079b220446dd85a5b40102742330 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/44df079b220446dd85a5b40102742330 2023-04-03 02:54:03,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680490442753, max timestamp=1680490443278Num uniques 1226; 2023-04-03 02:54:03,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=8056, suffix heap size=155472,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=176 2023-04-03 02:54:03,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680490442753, max timestamp=1680490443278Num uniques 1226; 2023-04-03 02:54:03,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=8056, suffix heap size=155472,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=176 2023-04-03 02:54:03,287 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680490442753, max timestamp=1680490443278Num uniques 1076; 2023-04-03 02:54:03,287 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-03 02:54:03,303 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/B of 459f839097be2d0597a0a35d5312fdc4 into 44df079b220446dd85a5b40102742330(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:03,303 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:03,303 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/B, priority=13, startTime=1680490443229; duration=0sec 2023-04-03 02:54:03,303 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:03,304 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:03,305 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:03,306 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/C is initiating minor compaction (all files) 2023-04-03 02:54:03,306 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/C in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,306 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a9b33cd7237b45fd8b3d5435984cf99b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/5c4d857408844a6ebd6f637a9846611d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/0c56e9b687d84095bddb40b1c9c37224] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=34.2 K 2023-04-03 02:54:03,307 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a9b33cd7237b45fd8b3d5435984cf99b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680490438958 2023-04-03 02:54:03,307 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 5c4d857408844a6ebd6f637a9846611d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1680490439579 2023-04-03 02:54:03,308 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 0c56e9b687d84095bddb40b1c9c37224, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1680490441209 2023-04-03 02:54:03,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:54:03,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=53.67 KB heapSize=144.33 KB 2023-04-03 02:54:03,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:54:03,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:03,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:54:03,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:03,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:54:03,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:03,319 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:03,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:03,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:03,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:03,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:03,333 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:03,333 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,333 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:03,333 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:03,334 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,334 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,334 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#C#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:03,335 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,335 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,336 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,336 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:03,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:03,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:03,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:03,339 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:03,339 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,339 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:03,348 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,349 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:03,349 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:03,349 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:03,352 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,352 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,353 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,353 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:03,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7353010a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:03,355 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:03,355 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,358 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,358 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:03,360 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:03,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:03,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:03,362 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:03,362 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,362 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,362 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:03,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:03,367 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:03,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:03,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:03,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:03,368 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:03,373 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,373 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,373 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,375 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,375 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/1bc55cf5906040fcb717296e852819bd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/1bc55cf5906040fcb717296e852819bd 2023-04-03 02:54:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,377 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,381 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,381 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,381 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,382 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/C of 459f839097be2d0597a0a35d5312fdc4 into 1bc55cf5906040fcb717296e852819bd(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:03,382 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:03,382 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/C, priority=13, startTime=1680490443229; duration=0sec 2023-04-03 02:54:03,383 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:03,383 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,383 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,383 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,383 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:03,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,384 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:03,384 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:03,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:03,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490443366, max timestamp=1680490443386Num uniques 859; 2023-04-03 02:54:03,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:54:03,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490443348, max timestamp=1680490443386Num uniques 859; 2023-04-03 02:54:03,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:54:03,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490443366, max timestamp=1680490443386Num uniques 859; 2023-04-03 02:54:03,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:54:03,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490503420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490503421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490503423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,484 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:03,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:03,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:03,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:03,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:03,488 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:03,488 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,488 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:03,488 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:03,488 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,488 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,490 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,490 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,490 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,490 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:03,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:03,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:03,491 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:03,491 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,491 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:03,492 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,492 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:03,492 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:03,492 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:03,493 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:03,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d7cd4[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:03,495 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:03,496 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,498 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,498 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:03,498 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:03,498 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:03,498 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,498 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:03,502 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,502 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:03,502 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,502 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:03,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:03,505 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:03,505 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:03,505 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:03,505 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:03,505 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:03,506 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,506 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,506 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,507 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,509 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:03,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:03,511 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,512 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,512 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:03,512 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:03,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:03,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490503524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490503525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490503527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,612 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:03,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:03,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:03,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:03,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:03,615 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:03,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:03,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:03,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,620 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,620 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,620 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,620 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:03,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:03,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:03,622 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:03,622 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:03,622 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,622 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:03,622 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:03,622 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,622 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:03,622 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:03,622 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,622 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:03,622 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:03,623 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,624 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,624 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:03,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bdc8faa[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:03,625 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:03,626 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:03,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:03,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:03,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:03,631 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,631 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,631 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,632 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:03,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:03,634 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:03,634 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:03,634 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:03,634 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:03,634 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:03,636 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,636 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,636 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,637 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,638 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,641 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:03,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:03,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:03,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,642 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:03,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,642 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:03,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,676 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/c801b4150474497ab380ba0d8dfd9952 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/c801b4150474497ab380ba0d8dfd9952 2023-04-03 02:54:03,682 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/A of 459f839097be2d0597a0a35d5312fdc4 into c801b4150474497ab380ba0d8dfd9952(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:03,682 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:03,682 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/A, priority=13, startTime=1680490443228; duration=0sec 2023-04-03 02:54:03,682 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:03,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47642 deadline: 1680490503715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,719 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:54:03,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47654 deadline: 1680490503722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,729 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:54:03,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490503733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490503733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:03,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490503733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,742 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:03,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:03,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:03,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:03,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:03,752 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:03,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:03,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:03,753 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,753 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,754 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,754 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,754 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,754 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:03,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:03,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:03,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,757 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:03,760 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,760 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:03,760 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:03,760 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:03,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,762 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,763 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,763 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:03,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55bb6760[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:03,764 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:03,764 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,767 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,767 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:03,767 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:03,767 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:03,767 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,767 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:03,769 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:03,769 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,769 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,770 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:03,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:03,773 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:03,773 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:03,773 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:03,773 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:03,773 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:03,774 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,774 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,775 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,777 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,779 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,779 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,779 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:03,780 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,780 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,780 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,780 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,780 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,780 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:03,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:03,780 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,780 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:03,780 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:03,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,785 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=929 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/03d1b73da1f14df6b487c988aa33d0e5 2023-04-03 02:54:03,880 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:03,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:03,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:03,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:03,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:03,884 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:03,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:03,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:03,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,886 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:03,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:03,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:03,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:03,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:03,889 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,889 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:03,889 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:03,889 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:03,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,892 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,892 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,892 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:03,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:03,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e0b678c[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:03,894 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:03,894 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,895 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,896 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:03,896 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:03,896 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:03,896 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,896 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:03,897 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,897 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:03,897 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,897 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:03,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:03,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:03,900 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:03,900 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:03,900 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:03,900 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:03,900 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:03,902 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,902 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:03,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:03,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,902 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:03,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:03,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,903 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:03,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:03,906 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:03,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,913 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,913 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:03,913 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:03,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:03,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:03,913 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:03,913 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:03,913 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:03,913 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:03,913 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:03,913 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:03,913 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,913 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:03,913 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:03,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:04,013 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:04,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:04,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:04,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:04,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:04,017 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:04,017 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,017 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:04,017 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:04,017 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,017 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,018 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,018 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:04,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:04,019 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,019 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:04,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:04,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:04,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:04,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:04,020 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:04,020 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,020 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:04,020 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:04,020 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:04,020 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:04,020 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:04,020 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:04,020 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:04,020 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:04,020 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:04,021 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,021 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:04,021 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:04,022 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,022 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,022 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,022 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:04,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@529cd745[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:04,028 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:04,028 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,034 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,034 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,034 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:04,034 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:04,034 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:04,034 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:04,036 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,036 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:04,036 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:04,036 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:04,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490504036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:04,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490504036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:04,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490504036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:04,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:04,040 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:04,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:04,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:04,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:04,040 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:04,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:04,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,044 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,045 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,048 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,048 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,048 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,048 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:04,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:04,048 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:04,049 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:04,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:04,048 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:04,049 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,049 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:04,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:04,049 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:04,049 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:04,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:04,149 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:04,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:04,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:04,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:04,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:04,154 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:04,154 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,154 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:04,154 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:04,155 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,155 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,157 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:04,157 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:04,158 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,158 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:04,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:04,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:04,159 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:04,159 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,159 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:04,159 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:04,159 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:04,159 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:04,159 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:04,160 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:04,160 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:04,161 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:04,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:04,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:04,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:04,164 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,164 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,165 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,165 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:04,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d12a363[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:04,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:04,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,188 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,188 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:04,188 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:04,188 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,188 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:04,188 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:04,190 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:04,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:04,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:04,194 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:04,194 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:04,194 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:04,194 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:04,194 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:04,197 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,197 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:04,197 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:04,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:04,198 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,199 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,202 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,202 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,202 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:04,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:04,202 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:04,202 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:04,202 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,203 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:04,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:04,202 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:04,203 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,203 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,203 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:04,203 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:04,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=929 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/f2e9789028bd41deb67b5644e3acd887 2023-04-03 02:54:04,303 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:04,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:04,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:04,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:04,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:04,316 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:04,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:04,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:04,317 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,317 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,319 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:04,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:04,319 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:04,320 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,320 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:04,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:04,337 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:04,337 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,337 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:04,344 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:04,344 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:04,344 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:04,344 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:04,344 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:04,344 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:04,344 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:04,344 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:04,344 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,344 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:04,344 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:04,346 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,346 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,346 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,346 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:04,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:04,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731c4bc6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:04,348 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:04,348 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,350 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,350 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:04,350 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:04,350 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:04,350 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,350 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:04,351 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:04,351 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,351 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:04,352 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:04,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:04,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:04,356 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:04,356 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:04,356 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:04,357 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:04,357 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:04,358 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,358 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:04,358 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:04,359 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:04,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:04,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:04,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,375 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:04,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:04,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:04,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:04,375 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:04,375 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:04,476 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:04,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:04,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:04,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:04,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:04,479 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:04,479 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,479 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:04,479 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:04,480 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,480 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,483 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:04,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:04,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:04,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:04,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:04,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:04,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:04,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:04,485 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:04,485 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:04,485 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:04,485 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:04,487 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,487 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,487 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,487 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:04,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ce652e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:04,489 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:04,489 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,491 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:04,492 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,492 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:04,493 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:04,493 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:04,493 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,493 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:04,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:04,496 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:04,496 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:04,496 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:04,496 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:04,496 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:04,498 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,498 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,498 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,499 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,500 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:04,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:04,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:04,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:04,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,504 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:04,504 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:04,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:04,504 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:04,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:04,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:04,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:04,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490504539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:04,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490504542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:04,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490504542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,604 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:04,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:04,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:04,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:04,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:04,608 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:04,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:04,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:04,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,611 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,611 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:04,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:04,611 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,611 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:04,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:04,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:04,613 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:04,613 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,613 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:04,613 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:04,613 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:04,614 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:04,614 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:04,614 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:04,614 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:04,614 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:04,614 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:04,614 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,614 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:04,614 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:04,615 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,616 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,616 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:04,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@225cbe4d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:04,617 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:04,617 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,618 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,619 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:04,619 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:04,619 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:04,619 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,619 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:04,620 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:04,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:04,622 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:04,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:04,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:04,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:04,623 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:04,625 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,625 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,625 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:04,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:04,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:04,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:04,626 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:04,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:04,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,628 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:04,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:04,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:04,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:04,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:04,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:04,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:04,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,630 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:04,631 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:04,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:04,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:04,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=929 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/fd154eede4a64f9e8752e62c273708c7 2023-04-03 02:54:04,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/03d1b73da1f14df6b487c988aa33d0e5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/03d1b73da1f14df6b487c988aa33d0e5 2023-04-03 02:54:04,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/03d1b73da1f14df6b487c988aa33d0e5, entries=150, sequenceid=929, filesize=11.9 K 2023-04-03 02:54:04,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/f2e9789028bd41deb67b5644e3acd887 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f2e9789028bd41deb67b5644e3acd887 2023-04-03 02:54:04,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f2e9789028bd41deb67b5644e3acd887, entries=150, sequenceid=929, filesize=11.9 K 2023-04-03 02:54:04,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/fd154eede4a64f9e8752e62c273708c7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/fd154eede4a64f9e8752e62c273708c7 2023-04-03 02:54:04,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/fd154eede4a64f9e8752e62c273708c7, entries=150, sequenceid=929, filesize=11.9 K 2023-04-03 02:54:04,697 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~144.28 KB/147744, currentSize=136.42 KB/139690 for 459f839097be2d0597a0a35d5312fdc4 in 1386ms, sequenceid=929, compaction requested=false 2023-04-03 02:54:04,697 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:04,731 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:04,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:04,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:04,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:04,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:04,736 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:04,736 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,736 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:04,736 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:04,736 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,736 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,738 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:04,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:04,738 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,738 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:04,738 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:04,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:04,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:04,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:04,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:04,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:04,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:04,739 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:04,739 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:04,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:04,744 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:04,744 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:04,744 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:04,744 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:04,744 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=136.42 KB heapSize=385.38 KB 2023-04-03 02:54:04,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:54:04,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:04,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:54:04,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:04,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:54:04,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:04,758 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/5b1c8876652f4c9f8a093e67eef54ca7 2023-04-03 02:54:05,194 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/953ad62de705461ea526e8549c422822 2023-04-03 02:54:05,213 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/021993f43cc44eea889c1f3a9ff83313 2023-04-03 02:54:05,219 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/5b1c8876652f4c9f8a093e67eef54ca7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5b1c8876652f4c9f8a093e67eef54ca7 2023-04-03 02:54:05,224 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5b1c8876652f4c9f8a093e67eef54ca7, entries=150, sequenceid=967, filesize=11.9 K 2023-04-03 02:54:05,225 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/953ad62de705461ea526e8549c422822 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/953ad62de705461ea526e8549c422822 2023-04-03 02:54:05,231 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/953ad62de705461ea526e8549c422822, entries=150, sequenceid=967, filesize=11.9 K 2023-04-03 02:54:05,232 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/021993f43cc44eea889c1f3a9ff83313 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/021993f43cc44eea889c1f3a9ff83313 2023-04-03 02:54:05,237 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/021993f43cc44eea889c1f3a9ff83313, entries=150, sequenceid=967, filesize=11.9 K 2023-04-03 02:54:05,238 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~385.33 KB/394576, currentSize=0 B/0 for 459f839097be2d0597a0a35d5312fdc4 in 494ms, sequenceid=967, compaction requested=true 2023-04-03 02:54:05,238 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:05,238 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:05,238 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:05,240 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:05,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4abd18de[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:05,243 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:05,243 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:05,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:05,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:05,244 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,244 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:05,245 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:05,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:05,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:05,248 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:05,248 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:05,248 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:05,249 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:05,249 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:05,251 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,251 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,251 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:05,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,252 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,259 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:05,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:05,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:05,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:05,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:05,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:05,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:05,269 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:05,269 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:05,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:05,369 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:05,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:05,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:05,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:05,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:05,373 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:05,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:05,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:05,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:05,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,378 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,378 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:05,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:05,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:05,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,379 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,379 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:05,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:05,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:05,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:05,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:05,386 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:05,386 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:05,387 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:05,387 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:05,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:05,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:05,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:05,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:05,389 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:05,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bee2834[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:05,394 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:05,394 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,395 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,395 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:05,395 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:05,395 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:05,395 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,395 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:05,397 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:05,397 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:05,397 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:05,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:05,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:05,401 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:05,401 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:05,401 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:05,401 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:05,401 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:05,403 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,404 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,404 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:05,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,404 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:05,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:05,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:05,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:05,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:05,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:05,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,410 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,410 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,410 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,409 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:05,410 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:05,410 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:05,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:05,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:05,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:05,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:05,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:05,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:05,515 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:05,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:05,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:05,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:05,516 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,516 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,517 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:05,517 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:05,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:05,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:05,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:05,522 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:05,522 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:05,522 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:05,523 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:05,523 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:05,523 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:05,523 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:05,524 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:05,524 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:05,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:05,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:05,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:05,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:05,525 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,525 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,526 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,526 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:05,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6cd172[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:05,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:05,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,532 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:05,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:05,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:05,532 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,532 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:05,533 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:05,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:05,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:05,538 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:05,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:05,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:05,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:05,538 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:05,540 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,540 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,540 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:05,541 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,542 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:05,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:05,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:05,545 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:05,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:05,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:05,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:05,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,545 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:05,545 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:05,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,546 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:05,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:54:05,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:54:05,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:54:05,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:05,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:54:05,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:05,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:54:05,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:05,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680490445600, max timestamp=1680490445610Num uniques 859; 2023-04-03 02:54:05,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 02:54:05,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680490445600, max timestamp=1680490445610Num uniques 859; 2023-04-03 02:54:05,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680490445600, max timestamp=1680490445610Num uniques 859; 2023-04-03 02:54:05,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 02:54:05,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 02:54:05,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:05,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490505639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:05,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490505639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:05,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490505640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,646 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:05,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:05,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:05,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:05,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:05,652 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:05,652 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:05,652 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:05,652 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:05,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,655 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:05,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:05,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:05,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:05,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:05,665 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:05,665 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:05,665 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:05,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:05,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:05,687 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:05,687 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:05,687 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:05,687 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:05,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:05,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:05,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:05,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:05,689 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,689 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,690 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,690 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:05,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7985ae24[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:05,693 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:05,693 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,695 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:05,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:05,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:05,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:05,697 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:05,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:05,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:05,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:05,700 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:05,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:05,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:05,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:05,700 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:05,704 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,704 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:05,704 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,709 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,712 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:05,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:05,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:05,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:05,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:05,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:05,716 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:05,716 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:05,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,717 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,717 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,717 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,717 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,716 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:05,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:05,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490505746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:05,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490505746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:05,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490505747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,817 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:05,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:05,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:05,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:05,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:05,822 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:05,822 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:05,822 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:05,822 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:05,822 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,822 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,823 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,823 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:05,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:05,824 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,824 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:05,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:05,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:05,825 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:05,825 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:05,825 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:05,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:05,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:05,826 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:05,826 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:05,826 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:05,826 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:05,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:05,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:05,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:05,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:05,828 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,828 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:05,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b5c3222[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:05,831 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:05,831 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,840 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,840 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:05,840 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:05,840 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:05,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:05,841 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:05,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,841 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,841 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:05,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:05,845 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:05,845 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:05,845 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:05,845 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:05,845 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:05,847 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,847 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:05,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:05,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,847 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:05,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:05,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:05,848 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:05,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,849 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:05,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:05,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:05,851 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:05,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:05,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:05,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:05,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:05,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:05,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:05,852 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:05,852 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:05,952 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:05,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:05,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:05,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490505951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:05,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490505952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:05,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490505952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:05,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:05,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:05,957 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:05,957 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:05,957 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:05,957 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:05,960 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:52506 2023-04-03 02:54:05,960 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:05,962 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:05,962 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:05,962 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d6b6ea to 127.0.0.1:52506 2023-04-03 02:54:05,962 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:05,964 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:52506 2023-04-03 02:54:05,964 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:05,964 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:05,965 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:05,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:05,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:05,969 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63871a9d to 127.0.0.1:52506 2023-04-03 02:54:05,969 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:05,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=980 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/f941279bbf98439d8bd65258cd4cf5b2 2023-04-03 02:54:06,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:06,069 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,069 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:06,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:06,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:06,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:06,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:06,070 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:06,070 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:06,070 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:06,070 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:06,070 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:06,070 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:06,071 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. started... 2023-04-03 02:54:06,071 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. as already flushing 2023-04-03 02:54:06,071 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:06,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:06,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:06,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:06,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:06,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:06,072 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:06,073 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:06,073 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:06,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:06,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d90ef5[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:06,074 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:06,074 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:06,075 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:06,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:06,075 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:06,075 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:06,075 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:06,075 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,076 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:06,077 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:06,077 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:06,077 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,077 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:06,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:06,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:06,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:06,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:06,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:06,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:06,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:06,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:06,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:06,079 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:06,079 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:06,079 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:06,079 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:06,079 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:06,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:06,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:06,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:06,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:06,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:06,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:06,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:06,083 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:06,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:06,084 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:06,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:06,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,086 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:06,086 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:06,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:06,086 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:06,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:06,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:06,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:06,086 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:06,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:06,086 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:06,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:06,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:06,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:06,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:06,087 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:06,087 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:06,087 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:06,187 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:06,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:06,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:06,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490506255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:06,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490506255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:06,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490506257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,383 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=980 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/d7b1a35d55ec4480bed9e54ce57bbc84 2023-04-03 02:54:06,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:06,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47586 deadline: 1680490506757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:06,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47626 deadline: 1680490506758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:06,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47670 deadline: 1680490506761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:06,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=980 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/18c6dfde42bc4333bdeaa68265c15e72 2023-04-03 02:54:06,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/f941279bbf98439d8bd65258cd4cf5b2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f941279bbf98439d8bd65258cd4cf5b2 2023-04-03 02:54:06,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f941279bbf98439d8bd65258cd4cf5b2, entries=250, sequenceid=980, filesize=16.7 K 2023-04-03 02:54:06,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/d7b1a35d55ec4480bed9e54ce57bbc84 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d7b1a35d55ec4480bed9e54ce57bbc84 2023-04-03 02:54:06,807 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d7b1a35d55ec4480bed9e54ce57bbc84, entries=150, sequenceid=980, filesize=11.9 K 2023-04-03 02:54:06,808 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/18c6dfde42bc4333bdeaa68265c15e72 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/18c6dfde42bc4333bdeaa68265c15e72 2023-04-03 02:54:06,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/18c6dfde42bc4333bdeaa68265c15e72, entries=150, sequenceid=980, filesize=11.9 K 2023-04-03 02:54:06,812 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 459f839097be2d0597a0a35d5312fdc4 in 1258ms, sequenceid=980, compaction requested=true 2023-04-03 02:54:06,812 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:06,813 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:06,813 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:54:06,813 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:54:06,813 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:06,813 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:06,814 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54680 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:54:06,814 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/A is initiating minor compaction (all files) 2023-04-03 02:54:06,814 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:54:06,814 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/A in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:06,814 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/B is initiating minor compaction (all files) 2023-04-03 02:54:06,814 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/c801b4150474497ab380ba0d8dfd9952, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/03d1b73da1f14df6b487c988aa33d0e5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5b1c8876652f4c9f8a093e67eef54ca7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f941279bbf98439d8bd65258cd4cf5b2] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=53.4 K 2023-04-03 02:54:06,814 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/B in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:06,814 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/44df079b220446dd85a5b40102742330, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f2e9789028bd41deb67b5644e3acd887, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/953ad62de705461ea526e8549c422822, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d7b1a35d55ec4480bed9e54ce57bbc84] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=48.6 K 2023-04-03 02:54:06,815 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c801b4150474497ab380ba0d8dfd9952, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1680490441209 2023-04-03 02:54:06,815 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 44df079b220446dd85a5b40102742330, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1680490441209 2023-04-03 02:54:06,815 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 03d1b73da1f14df6b487c988aa33d0e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=929, earliestPutTs=1680490443292 2023-04-03 02:54:06,815 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f2e9789028bd41deb67b5644e3acd887, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=929, earliestPutTs=1680490443292 2023-04-03 02:54:06,815 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 5b1c8876652f4c9f8a093e67eef54ca7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1680490443406 2023-04-03 02:54:06,816 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 953ad62de705461ea526e8549c422822, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1680490443406 2023-04-03 02:54:06,816 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting f941279bbf98439d8bd65258cd4cf5b2, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=980, earliestPutTs=1680490445549 2023-04-03 02:54:06,816 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d7b1a35d55ec4480bed9e54ce57bbc84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=980, earliestPutTs=1680490445549 2023-04-03 02:54:06,825 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#A#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:06,827 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#B#compaction#106 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:07,237 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/28a9aaffbd2d4e708388eb964fb6eab0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/28a9aaffbd2d4e708388eb964fb6eab0 2023-04-03 02:54:07,240 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/2a5302e2009b4e09b38b2df772d86fe0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/2a5302e2009b4e09b38b2df772d86fe0 2023-04-03 02:54:07,243 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/A of 459f839097be2d0597a0a35d5312fdc4 into 28a9aaffbd2d4e708388eb964fb6eab0(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:07,243 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:07,243 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/A, priority=12, startTime=1680490446813; duration=0sec 2023-04-03 02:54:07,243 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:07,243 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:54:07,244 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:54:07,244 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 459f839097be2d0597a0a35d5312fdc4/C is initiating minor compaction (all files) 2023-04-03 02:54:07,244 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 459f839097be2d0597a0a35d5312fdc4/C in TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:07,245 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/1bc55cf5906040fcb717296e852819bd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/fd154eede4a64f9e8752e62c273708c7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/021993f43cc44eea889c1f3a9ff83313, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/18c6dfde42bc4333bdeaa68265c15e72] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp, totalSize=48.6 K 2023-04-03 02:54:07,245 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 1bc55cf5906040fcb717296e852819bd, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1680490441209 2023-04-03 02:54:07,245 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/B of 459f839097be2d0597a0a35d5312fdc4 into 2a5302e2009b4e09b38b2df772d86fe0(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:07,245 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:07,245 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/B, priority=12, startTime=1680490446813; duration=0sec 2023-04-03 02:54:07,245 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting fd154eede4a64f9e8752e62c273708c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=929, earliestPutTs=1680490443292 2023-04-03 02:54:07,245 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:07,246 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 021993f43cc44eea889c1f3a9ff83313, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1680490443406 2023-04-03 02:54:07,246 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 18c6dfde42bc4333bdeaa68265c15e72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=980, earliestPutTs=1680490445549 2023-04-03 02:54:07,257 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 459f839097be2d0597a0a35d5312fdc4#C#compaction#107 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:07,666 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/fa774792a7ba4cac8b8b08c68ad763d7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/fa774792a7ba4cac8b8b08c68ad763d7 2023-04-03 02:54:07,671 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 459f839097be2d0597a0a35d5312fdc4/C of 459f839097be2d0597a0a35d5312fdc4 into fa774792a7ba4cac8b8b08c68ad763d7(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:07,671 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:07,671 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4., storeName=459f839097be2d0597a0a35d5312fdc4/C, priority=12, startTime=1680490446813; duration=0sec 2023-04-03 02:54:07,671 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:07,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:54:07,744 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-03 02:54:07,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:54:07,745 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de4d1f6 to 127.0.0.1:52506 2023-04-03 02:54:07,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:07,745 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:07,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:54:07,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:07,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:54:07,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:07,758 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:52506 2023-04-03 02:54:07,758 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:07,767 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02c57e1b to 127.0.0.1:52506 2023-04-03 02:54:07,767 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:07,768 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:52506 2023-04-03 02:54:07,768 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:07,769 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56223d42 to 127.0.0.1:52506 2023-04-03 02:54:07,770 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 232 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 251 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 86 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 239 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 4058 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 3870 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1759 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5275 rows 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1763 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5284 rows 2023-04-03 02:54:07,770 INFO [Listener at localhost/46853] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 02:54:07,770 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08c3f619 to 127.0.0.1:52506 2023-04-03 02:54:07,771 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:07,776 INFO [Listener at localhost/46853] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 02:54:07,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 02:54:07,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:07,790 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490447790"}]},"ts":"1680490447790"} 2023-04-03 02:54:07,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 02:54:07,791 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 02:54:07,793 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 02:54:07,795 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=459f839097be2d0597a0a35d5312fdc4, UNASSIGN}] 2023-04-03 02:54:07,796 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=459f839097be2d0597a0a35d5312fdc4, UNASSIGN 2023-04-03 02:54:07,797 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=459f839097be2d0597a0a35d5312fdc4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:07,797 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490447797"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490447797"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490447797"}]},"ts":"1680490447797"} 2023-04-03 02:54:07,799 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:54:07,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 02:54:07,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:54:07,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 459f839097be2d0597a0a35d5312fdc4, disabling compactions & flushes 2023-04-03 02:54:07,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:08,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 02:54:08,153 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1018 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/5cec121ecae4434f8535d602f8c2dfd3 2023-04-03 02:54:08,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 02:54:08,565 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1018 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/b5ae93fc42ba409a8a83ca8151da3825 2023-04-03 02:54:08,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 02:54:08,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1018 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/0d31d6f2446f44b9a3031466e9ae28ee 2023-04-03 02:54:08,981 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/5cec121ecae4434f8535d602f8c2dfd3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5cec121ecae4434f8535d602f8c2dfd3 2023-04-03 02:54:08,985 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5cec121ecae4434f8535d602f8c2dfd3, entries=150, sequenceid=1018, filesize=11.9 K 2023-04-03 02:54:08,986 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/b5ae93fc42ba409a8a83ca8151da3825 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/b5ae93fc42ba409a8a83ca8151da3825 2023-04-03 02:54:08,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/b5ae93fc42ba409a8a83ca8151da3825, entries=150, sequenceid=1018, filesize=11.9 K 2023-04-03 02:54:08,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/0d31d6f2446f44b9a3031466e9ae28ee as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/0d31d6f2446f44b9a3031466e9ae28ee 2023-04-03 02:54:08,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/0d31d6f2446f44b9a3031466e9ae28ee, entries=150, sequenceid=1018, filesize=11.9 K 2023-04-03 02:54:08,996 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=26.84 KB/27480 for 459f839097be2d0597a0a35d5312fdc4 in 1252ms, sequenceid=1018, compaction requested=false 2023-04-03 02:54:08,996 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:08,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:08,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:08,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. after waiting 0 ms 2023-04-03 02:54:08,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:08,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 459f839097be2d0597a0a35d5312fdc4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 02:54:08,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=A 2023-04-03 02:54:08,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:08,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=B 2023-04-03 02:54:08,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:08,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 459f839097be2d0597a0a35d5312fdc4, store=C 2023-04-03 02:54:08,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:09,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1025 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/acd6325fc3d6468cac5582d38f6f3e46 2023-04-03 02:54:09,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1025 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/36032fe6b6b4434b952d2f86f2c01d19 2023-04-03 02:54:09,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 02:54:10,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1025 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/e65ad40b66c54ef8a95b845859ddef41 2023-04-03 02:54:10,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/A/acd6325fc3d6468cac5582d38f6f3e46 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/acd6325fc3d6468cac5582d38f6f3e46 2023-04-03 02:54:10,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/acd6325fc3d6468cac5582d38f6f3e46, entries=100, sequenceid=1025, filesize=9.5 K 2023-04-03 02:54:10,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/B/36032fe6b6b4434b952d2f86f2c01d19 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/36032fe6b6b4434b952d2f86f2c01d19 2023-04-03 02:54:10,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/36032fe6b6b4434b952d2f86f2c01d19, entries=100, sequenceid=1025, filesize=9.5 K 2023-04-03 02:54:10,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/.tmp/C/e65ad40b66c54ef8a95b845859ddef41 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/e65ad40b66c54ef8a95b845859ddef41 2023-04-03 02:54:10,245 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/e65ad40b66c54ef8a95b845859ddef41, entries=100, sequenceid=1025, filesize=9.5 K 2023-04-03 02:54:10,246 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 459f839097be2d0597a0a35d5312fdc4 in 1250ms, sequenceid=1025, compaction requested=true 2023-04-03 02:54:10,268 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a2d1e18d90ad443f858af2775e7aecf5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6ac3460e5021409ba0a77c82b449ea0b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/d7e7a5ac701b4693954707d7ecdd1d7a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/24425fbcc6de49888941d0ce35ed8af6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f36d710c700348c1a412ae8fcb2ae47c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/51508d851ed749c9b78f46b9fcdc93a5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/1e4c6968f7ed4f2689bd99e908db1fe8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/4355af2523024544a0658c51ca0c57e1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/61211523fd62432986ff5ef2c7e3e124, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/27f17e3b4bef4e478b338b5c23b7d537, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a96519779818495889cda5d0aee88da1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/cf320548934341e9bd4c1dd23b6b0ced, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/7ffafcf7eb464b1d846203a0ce50b041, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/adec6bc86a5a4839985a6a3acd9db52d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ba50c01bc97347e09f0c0afc1fb8c8b7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/bd6cde21a80443a18530a4a76930e484, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/672bc373f29747ad9ac8799be95cb5b5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5311b6d054e444e98025b949d0783a3f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/e7c5d0a716114315a1dbd0746d6993f7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/c0b589a652ec46f981a03287de16dc8a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f6f7b054a9fd4ca8adf4167670c8de51, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/e6596c3216174329aacd0121b9cbb2ab, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/3437d51aecad42dda50078e8dc47c1c0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6131c29d2aae4e7ba011e5a1a637aef2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/9c787892aa844ede82dbd198ba1d0b67, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/bc39edcad2cd4ab9be1f513bcd4fe523, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ff66486a9909429da7b4018b8d149148, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/0cf3f43bceb242e0bb4411c0bc745c82, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a9c24060c5b5491eb199319b6946995f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/746672cc61d64bdc9ba77474b953fa8b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ac7deb6c9adb4434a10fa50e65644aac, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/c801b4150474497ab380ba0d8dfd9952, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/03d1b73da1f14df6b487c988aa33d0e5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5b1c8876652f4c9f8a093e67eef54ca7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f941279bbf98439d8bd65258cd4cf5b2] to archive 2023-04-03 02:54:10,269 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:54:10,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a2d1e18d90ad443f858af2775e7aecf5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a2d1e18d90ad443f858af2775e7aecf5 2023-04-03 02:54:10,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6ac3460e5021409ba0a77c82b449ea0b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6ac3460e5021409ba0a77c82b449ea0b 2023-04-03 02:54:10,278 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/d7e7a5ac701b4693954707d7ecdd1d7a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/d7e7a5ac701b4693954707d7ecdd1d7a 2023-04-03 02:54:10,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/24425fbcc6de49888941d0ce35ed8af6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/24425fbcc6de49888941d0ce35ed8af6 2023-04-03 02:54:10,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f36d710c700348c1a412ae8fcb2ae47c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f36d710c700348c1a412ae8fcb2ae47c 2023-04-03 02:54:10,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/51508d851ed749c9b78f46b9fcdc93a5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/51508d851ed749c9b78f46b9fcdc93a5 2023-04-03 02:54:10,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/1e4c6968f7ed4f2689bd99e908db1fe8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/1e4c6968f7ed4f2689bd99e908db1fe8 2023-04-03 02:54:10,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/4355af2523024544a0658c51ca0c57e1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/4355af2523024544a0658c51ca0c57e1 2023-04-03 02:54:10,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/61211523fd62432986ff5ef2c7e3e124 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/61211523fd62432986ff5ef2c7e3e124 2023-04-03 02:54:10,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/27f17e3b4bef4e478b338b5c23b7d537 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/27f17e3b4bef4e478b338b5c23b7d537 2023-04-03 02:54:10,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a96519779818495889cda5d0aee88da1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a96519779818495889cda5d0aee88da1 2023-04-03 02:54:10,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/cf320548934341e9bd4c1dd23b6b0ced to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/cf320548934341e9bd4c1dd23b6b0ced 2023-04-03 02:54:10,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/7ffafcf7eb464b1d846203a0ce50b041 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/7ffafcf7eb464b1d846203a0ce50b041 2023-04-03 02:54:10,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/adec6bc86a5a4839985a6a3acd9db52d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/adec6bc86a5a4839985a6a3acd9db52d 2023-04-03 02:54:10,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ba50c01bc97347e09f0c0afc1fb8c8b7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ba50c01bc97347e09f0c0afc1fb8c8b7 2023-04-03 02:54:10,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/bd6cde21a80443a18530a4a76930e484 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/bd6cde21a80443a18530a4a76930e484 2023-04-03 02:54:10,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/672bc373f29747ad9ac8799be95cb5b5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/672bc373f29747ad9ac8799be95cb5b5 2023-04-03 02:54:10,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5311b6d054e444e98025b949d0783a3f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5311b6d054e444e98025b949d0783a3f 2023-04-03 02:54:10,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/e7c5d0a716114315a1dbd0746d6993f7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/e7c5d0a716114315a1dbd0746d6993f7 2023-04-03 02:54:10,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/c0b589a652ec46f981a03287de16dc8a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/c0b589a652ec46f981a03287de16dc8a 2023-04-03 02:54:10,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f6f7b054a9fd4ca8adf4167670c8de51 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f6f7b054a9fd4ca8adf4167670c8de51 2023-04-03 02:54:10,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/e6596c3216174329aacd0121b9cbb2ab to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/e6596c3216174329aacd0121b9cbb2ab 2023-04-03 02:54:10,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/3437d51aecad42dda50078e8dc47c1c0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/3437d51aecad42dda50078e8dc47c1c0 2023-04-03 02:54:10,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6131c29d2aae4e7ba011e5a1a637aef2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/6131c29d2aae4e7ba011e5a1a637aef2 2023-04-03 02:54:10,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/9c787892aa844ede82dbd198ba1d0b67 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/9c787892aa844ede82dbd198ba1d0b67 2023-04-03 02:54:10,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/bc39edcad2cd4ab9be1f513bcd4fe523 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/bc39edcad2cd4ab9be1f513bcd4fe523 2023-04-03 02:54:10,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ff66486a9909429da7b4018b8d149148 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ff66486a9909429da7b4018b8d149148 2023-04-03 02:54:10,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/0cf3f43bceb242e0bb4411c0bc745c82 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/0cf3f43bceb242e0bb4411c0bc745c82 2023-04-03 02:54:10,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a9c24060c5b5491eb199319b6946995f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/a9c24060c5b5491eb199319b6946995f 2023-04-03 02:54:10,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/746672cc61d64bdc9ba77474b953fa8b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/746672cc61d64bdc9ba77474b953fa8b 2023-04-03 02:54:10,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ac7deb6c9adb4434a10fa50e65644aac to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/ac7deb6c9adb4434a10fa50e65644aac 2023-04-03 02:54:10,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/c801b4150474497ab380ba0d8dfd9952 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/c801b4150474497ab380ba0d8dfd9952 2023-04-03 02:54:10,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/03d1b73da1f14df6b487c988aa33d0e5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/03d1b73da1f14df6b487c988aa33d0e5 2023-04-03 02:54:10,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5b1c8876652f4c9f8a093e67eef54ca7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5b1c8876652f4c9f8a093e67eef54ca7 2023-04-03 02:54:10,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f941279bbf98439d8bd65258cd4cf5b2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/f941279bbf98439d8bd65258cd4cf5b2 2023-04-03 02:54:10,337 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d42872c9839b4b008973d3deecd891fd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f21837a5e6234deb9202f5f7e9168457, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/8d582d34951c4f65a94f43a0516ec65d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/81fe91c87c3e46e298d563528d1f9418, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/05f40bd3d7d841389a021c8e7cd5c208, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/6f91eb59e52c49e29a9e9eb6747a3e71, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/26c9257ecb274c21918c782f09da35e2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/b4e43ad1591540baa83826b8d61b5754, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7db6361401e14aca854714822011a9a7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/9794d89cf4b54ddf807df2c1f61d8bb7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d0b6d445fb0643c8aa8214e827b5011a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/064538efcd5748b7a0142381d63241d6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/a714f56a0fbf413292acdc8bde4d91b7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/922cc7f7b3c44156b92687a8a4e72610, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/efb1f87d9f4b498488ef024932118d4e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/5eaf3075220841edaf559be32e9aba27, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/1d6a2bad49124b73a6f80466fe047cf4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/cfd82e03e3604f41929cf7b0e10848e7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/eefb5b77d22b4edc875aeaf0ae832d03, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/813f4ed329284119adcfb9a4d49aa910, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/802683618e5445eb9bcb972c2568661e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/db8ca8b74f9b471ea434c7b89f082ce3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/fc0c7bb99ae240b2987d9b13f3ca0b6b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7f700caa81ae4df1b085e9b354306645, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/32e08ecc422f4aee8945d1084df12aa9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/33fcd00506d542e2bd82a648e47ea2ee, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/0789239519864c75be770feba2bbee83, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/0771ed9e2d144e53aeb8621550edc906, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/41d4d4b60c094f7aa911b184d37bfa6f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/8487fe51e9aa44ad8af6931df091ab88, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/44df079b220446dd85a5b40102742330, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/567cc6a8d18b4e1f8e919776e96b62f8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f2e9789028bd41deb67b5644e3acd887, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/953ad62de705461ea526e8549c422822, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d7b1a35d55ec4480bed9e54ce57bbc84] to archive 2023-04-03 02:54:10,338 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:54:10,340 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d42872c9839b4b008973d3deecd891fd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d42872c9839b4b008973d3deecd891fd 2023-04-03 02:54:10,341 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f21837a5e6234deb9202f5f7e9168457 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f21837a5e6234deb9202f5f7e9168457 2023-04-03 02:54:10,342 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/8d582d34951c4f65a94f43a0516ec65d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/8d582d34951c4f65a94f43a0516ec65d 2023-04-03 02:54:10,343 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/81fe91c87c3e46e298d563528d1f9418 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/81fe91c87c3e46e298d563528d1f9418 2023-04-03 02:54:10,344 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/05f40bd3d7d841389a021c8e7cd5c208 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/05f40bd3d7d841389a021c8e7cd5c208 2023-04-03 02:54:10,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/6f91eb59e52c49e29a9e9eb6747a3e71 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/6f91eb59e52c49e29a9e9eb6747a3e71 2023-04-03 02:54:10,346 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/26c9257ecb274c21918c782f09da35e2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/26c9257ecb274c21918c782f09da35e2 2023-04-03 02:54:10,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/b4e43ad1591540baa83826b8d61b5754 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/b4e43ad1591540baa83826b8d61b5754 2023-04-03 02:54:10,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7db6361401e14aca854714822011a9a7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7db6361401e14aca854714822011a9a7 2023-04-03 02:54:10,349 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/9794d89cf4b54ddf807df2c1f61d8bb7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/9794d89cf4b54ddf807df2c1f61d8bb7 2023-04-03 02:54:10,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d0b6d445fb0643c8aa8214e827b5011a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d0b6d445fb0643c8aa8214e827b5011a 2023-04-03 02:54:10,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/064538efcd5748b7a0142381d63241d6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/064538efcd5748b7a0142381d63241d6 2023-04-03 02:54:10,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/a714f56a0fbf413292acdc8bde4d91b7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/a714f56a0fbf413292acdc8bde4d91b7 2023-04-03 02:54:10,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/922cc7f7b3c44156b92687a8a4e72610 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/922cc7f7b3c44156b92687a8a4e72610 2023-04-03 02:54:10,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/efb1f87d9f4b498488ef024932118d4e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/efb1f87d9f4b498488ef024932118d4e 2023-04-03 02:54:10,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/5eaf3075220841edaf559be32e9aba27 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/5eaf3075220841edaf559be32e9aba27 2023-04-03 02:54:10,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/1d6a2bad49124b73a6f80466fe047cf4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/1d6a2bad49124b73a6f80466fe047cf4 2023-04-03 02:54:10,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/cfd82e03e3604f41929cf7b0e10848e7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/cfd82e03e3604f41929cf7b0e10848e7 2023-04-03 02:54:10,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/eefb5b77d22b4edc875aeaf0ae832d03 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/eefb5b77d22b4edc875aeaf0ae832d03 2023-04-03 02:54:10,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/813f4ed329284119adcfb9a4d49aa910 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/813f4ed329284119adcfb9a4d49aa910 2023-04-03 02:54:10,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/802683618e5445eb9bcb972c2568661e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/802683618e5445eb9bcb972c2568661e 2023-04-03 02:54:10,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/db8ca8b74f9b471ea434c7b89f082ce3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/db8ca8b74f9b471ea434c7b89f082ce3 2023-04-03 02:54:10,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/fc0c7bb99ae240b2987d9b13f3ca0b6b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/fc0c7bb99ae240b2987d9b13f3ca0b6b 2023-04-03 02:54:10,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7f700caa81ae4df1b085e9b354306645 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/7f700caa81ae4df1b085e9b354306645 2023-04-03 02:54:10,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/32e08ecc422f4aee8945d1084df12aa9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/32e08ecc422f4aee8945d1084df12aa9 2023-04-03 02:54:10,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/33fcd00506d542e2bd82a648e47ea2ee to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/33fcd00506d542e2bd82a648e47ea2ee 2023-04-03 02:54:10,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/0789239519864c75be770feba2bbee83 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/0789239519864c75be770feba2bbee83 2023-04-03 02:54:10,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/0771ed9e2d144e53aeb8621550edc906 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/0771ed9e2d144e53aeb8621550edc906 2023-04-03 02:54:10,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/41d4d4b60c094f7aa911b184d37bfa6f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/41d4d4b60c094f7aa911b184d37bfa6f 2023-04-03 02:54:10,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/8487fe51e9aa44ad8af6931df091ab88 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/8487fe51e9aa44ad8af6931df091ab88 2023-04-03 02:54:10,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/44df079b220446dd85a5b40102742330 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/44df079b220446dd85a5b40102742330 2023-04-03 02:54:10,370 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/567cc6a8d18b4e1f8e919776e96b62f8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/567cc6a8d18b4e1f8e919776e96b62f8 2023-04-03 02:54:10,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f2e9789028bd41deb67b5644e3acd887 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/f2e9789028bd41deb67b5644e3acd887 2023-04-03 02:54:10,372 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/953ad62de705461ea526e8549c422822 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/953ad62de705461ea526e8549c422822 2023-04-03 02:54:10,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d7b1a35d55ec4480bed9e54ce57bbc84 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/d7b1a35d55ec4480bed9e54ce57bbc84 2023-04-03 02:54:10,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477638a8854f4ae387fade412079519f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3423eedc424d461f8acea47ccd2c5644, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/ea9bb2adc9d8464899dd80b7e0d47ef0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/1b762b3d8ea4437db9bb08effbc0e808, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/96251e9e770643eab525d12986b439cf, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3884fcd1c157430d9ba21117436d657d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/268736ad52b54e8f8f17019c4a3d2ba0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/846ee0010ad34d66b92db368c691b2dd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477f50be5c0543629650cb4c3e7fa899, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a9f94bd60d54437090c6103fdf8c1b42, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/5e55053cc5414204aba6c71b8039f747, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/45587ca221504628818f39ba905f60ec, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/db13386f44ce4988b7657395b8ef482d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a764a7f1e8ed41d7940ddc551dcb4826, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/28dba09954784eeb923581296ea60bd8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/ef39edd961634ee38aa7cd48df4dcf08, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/2e3bfbeba47a4868ba61853e906e0f51, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/4319afea2ee346c0a0d8ad34d549101f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6fa17bcf9e114c4d849f8b25ae9d7e56, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/b9291a29d2b647679f89771923cb0f5f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/f12cdb31fb504ce0a0be3f54d7c949f6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/91bfee7fd70f4a3b8a39d0bc9c00bc83, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/037c79c2fad045d7a75c4a8eb503b2d4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/9294ba4c3e2b4183b9d627a37768b857, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/7567630bbb1d43edafd5a2202de9ec4c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3c38d2a48f34445ca209c9303227a4d9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6424c06368c0416d834ae08f16eeb92e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a9b33cd7237b45fd8b3d5435984cf99b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/71db2c88178242f7924bd6bd1a7f8c58, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/5c4d857408844a6ebd6f637a9846611d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/1bc55cf5906040fcb717296e852819bd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/0c56e9b687d84095bddb40b1c9c37224, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/fd154eede4a64f9e8752e62c273708c7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/021993f43cc44eea889c1f3a9ff83313, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/18c6dfde42bc4333bdeaa68265c15e72] to archive 2023-04-03 02:54:10,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:54:10,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477638a8854f4ae387fade412079519f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477638a8854f4ae387fade412079519f 2023-04-03 02:54:10,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3423eedc424d461f8acea47ccd2c5644 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3423eedc424d461f8acea47ccd2c5644 2023-04-03 02:54:10,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/ea9bb2adc9d8464899dd80b7e0d47ef0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/ea9bb2adc9d8464899dd80b7e0d47ef0 2023-04-03 02:54:10,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/1b762b3d8ea4437db9bb08effbc0e808 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/1b762b3d8ea4437db9bb08effbc0e808 2023-04-03 02:54:10,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/96251e9e770643eab525d12986b439cf to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/96251e9e770643eab525d12986b439cf 2023-04-03 02:54:10,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3884fcd1c157430d9ba21117436d657d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3884fcd1c157430d9ba21117436d657d 2023-04-03 02:54:10,385 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/268736ad52b54e8f8f17019c4a3d2ba0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/268736ad52b54e8f8f17019c4a3d2ba0 2023-04-03 02:54:10,386 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/846ee0010ad34d66b92db368c691b2dd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/846ee0010ad34d66b92db368c691b2dd 2023-04-03 02:54:10,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477f50be5c0543629650cb4c3e7fa899 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/477f50be5c0543629650cb4c3e7fa899 2023-04-03 02:54:10,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a9f94bd60d54437090c6103fdf8c1b42 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a9f94bd60d54437090c6103fdf8c1b42 2023-04-03 02:54:10,388 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/5e55053cc5414204aba6c71b8039f747 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/5e55053cc5414204aba6c71b8039f747 2023-04-03 02:54:10,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/45587ca221504628818f39ba905f60ec to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/45587ca221504628818f39ba905f60ec 2023-04-03 02:54:10,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/db13386f44ce4988b7657395b8ef482d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/db13386f44ce4988b7657395b8ef482d 2023-04-03 02:54:10,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a764a7f1e8ed41d7940ddc551dcb4826 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a764a7f1e8ed41d7940ddc551dcb4826 2023-04-03 02:54:10,392 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/28dba09954784eeb923581296ea60bd8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/28dba09954784eeb923581296ea60bd8 2023-04-03 02:54:10,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/ef39edd961634ee38aa7cd48df4dcf08 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/ef39edd961634ee38aa7cd48df4dcf08 2023-04-03 02:54:10,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/2e3bfbeba47a4868ba61853e906e0f51 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/2e3bfbeba47a4868ba61853e906e0f51 2023-04-03 02:54:10,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/4319afea2ee346c0a0d8ad34d549101f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/4319afea2ee346c0a0d8ad34d549101f 2023-04-03 02:54:10,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6fa17bcf9e114c4d849f8b25ae9d7e56 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6fa17bcf9e114c4d849f8b25ae9d7e56 2023-04-03 02:54:10,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/b9291a29d2b647679f89771923cb0f5f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/b9291a29d2b647679f89771923cb0f5f 2023-04-03 02:54:10,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/f12cdb31fb504ce0a0be3f54d7c949f6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/f12cdb31fb504ce0a0be3f54d7c949f6 2023-04-03 02:54:10,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/91bfee7fd70f4a3b8a39d0bc9c00bc83 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/91bfee7fd70f4a3b8a39d0bc9c00bc83 2023-04-03 02:54:10,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/037c79c2fad045d7a75c4a8eb503b2d4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/037c79c2fad045d7a75c4a8eb503b2d4 2023-04-03 02:54:10,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/9294ba4c3e2b4183b9d627a37768b857 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/9294ba4c3e2b4183b9d627a37768b857 2023-04-03 02:54:10,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/7567630bbb1d43edafd5a2202de9ec4c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/7567630bbb1d43edafd5a2202de9ec4c 2023-04-03 02:54:10,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3c38d2a48f34445ca209c9303227a4d9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/3c38d2a48f34445ca209c9303227a4d9 2023-04-03 02:54:10,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6424c06368c0416d834ae08f16eeb92e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/6424c06368c0416d834ae08f16eeb92e 2023-04-03 02:54:10,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a9b33cd7237b45fd8b3d5435984cf99b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/a9b33cd7237b45fd8b3d5435984cf99b 2023-04-03 02:54:10,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/71db2c88178242f7924bd6bd1a7f8c58 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/71db2c88178242f7924bd6bd1a7f8c58 2023-04-03 02:54:10,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/5c4d857408844a6ebd6f637a9846611d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/5c4d857408844a6ebd6f637a9846611d 2023-04-03 02:54:10,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/1bc55cf5906040fcb717296e852819bd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/1bc55cf5906040fcb717296e852819bd 2023-04-03 02:54:10,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/0c56e9b687d84095bddb40b1c9c37224 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/0c56e9b687d84095bddb40b1c9c37224 2023-04-03 02:54:10,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/fd154eede4a64f9e8752e62c273708c7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/fd154eede4a64f9e8752e62c273708c7 2023-04-03 02:54:10,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/021993f43cc44eea889c1f3a9ff83313 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/021993f43cc44eea889c1f3a9ff83313 2023-04-03 02:54:10,414 DEBUG [StoreCloser-TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/18c6dfde42bc4333bdeaa68265c15e72 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/18c6dfde42bc4333bdeaa68265c15e72 2023-04-03 02:54:10,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/recovered.edits/1028.seqid, newMaxSeqId=1028, maxSeqId=1 2023-04-03 02:54:10,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4. 2023-04-03 02:54:10,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 459f839097be2d0597a0a35d5312fdc4: 2023-04-03 02:54:10,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:54:10,423 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=459f839097be2d0597a0a35d5312fdc4, regionState=CLOSED 2023-04-03 02:54:10,423 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490450423"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490450423"}]},"ts":"1680490450423"} 2023-04-03 02:54:10,426 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-03 02:54:10,426 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 459f839097be2d0597a0a35d5312fdc4, server=jenkins-hbase4.apache.org,43815,1680490421348 in 2.6250 sec 2023-04-03 02:54:10,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-03 02:54:10,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=459f839097be2d0597a0a35d5312fdc4, UNASSIGN in 2.6310 sec 2023-04-03 02:54:10,428 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490450428"}]},"ts":"1680490450428"} 2023-04-03 02:54:10,429 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 02:54:10,438 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 02:54:10,440 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.6570 sec 2023-04-03 02:54:11,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 02:54:11,896 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-03 02:54:11,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 02:54:11,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:11,910 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:11,911 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:11,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-03 02:54:11,917 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:54:11,921 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/recovered.edits] 2023-04-03 02:54:11,925 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/28a9aaffbd2d4e708388eb964fb6eab0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/28a9aaffbd2d4e708388eb964fb6eab0 2023-04-03 02:54:11,926 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5cec121ecae4434f8535d602f8c2dfd3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/5cec121ecae4434f8535d602f8c2dfd3 2023-04-03 02:54:11,927 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/acd6325fc3d6468cac5582d38f6f3e46 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/A/acd6325fc3d6468cac5582d38f6f3e46 2023-04-03 02:54:11,929 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/2a5302e2009b4e09b38b2df772d86fe0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/2a5302e2009b4e09b38b2df772d86fe0 2023-04-03 02:54:11,930 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/36032fe6b6b4434b952d2f86f2c01d19 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/36032fe6b6b4434b952d2f86f2c01d19 2023-04-03 02:54:11,931 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/b5ae93fc42ba409a8a83ca8151da3825 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/B/b5ae93fc42ba409a8a83ca8151da3825 2023-04-03 02:54:11,934 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/0d31d6f2446f44b9a3031466e9ae28ee to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/0d31d6f2446f44b9a3031466e9ae28ee 2023-04-03 02:54:11,935 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/e65ad40b66c54ef8a95b845859ddef41 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/e65ad40b66c54ef8a95b845859ddef41 2023-04-03 02:54:11,936 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/fa774792a7ba4cac8b8b08c68ad763d7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/C/fa774792a7ba4cac8b8b08c68ad763d7 2023-04-03 02:54:11,938 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/recovered.edits/1028.seqid to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4/recovered.edits/1028.seqid 2023-04-03 02:54:11,939 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/459f839097be2d0597a0a35d5312fdc4 2023-04-03 02:54:11,939 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:54:11,941 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:11,962 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 02:54:12,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-03 02:54:12,015 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 02:54:12,016 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:12,016 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 02:54:12,017 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680490452016"}]},"ts":"9223372036854775807"} 2023-04-03 02:54:12,019 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 02:54:12,019 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 459f839097be2d0597a0a35d5312fdc4, NAME => 'TestAcidGuarantees,,1680490424703.459f839097be2d0597a0a35d5312fdc4.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 02:54:12,019 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 02:54:12,019 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680490452019"}]},"ts":"9223372036854775807"} 2023-04-03 02:54:12,021 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 02:54:12,023 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:12,024 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 121 msec 2023-04-03 02:54:12,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-03 02:54:12,215 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-03 02:54:12,231 INFO [Listener at localhost/46853] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=233 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-294374796_15 at /127.0.0.1:41356 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:43815-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7564d4f3-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7564d4f3-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-294374796_15 at /127.0.0.1:41404 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7564d4f3-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7564d4f3-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=551 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=455 (was 442) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=3765 (was 4337) 2023-04-03 02:54:12,243 INFO [Listener at localhost/46853] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=233, OpenFileDescriptor=551, MaxFileDescriptor=60000, SystemLoadAverage=455, ProcessCount=170, AvailableMemoryMB=3765 2023-04-03 02:54:12,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 02:54:12,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 02:54:12,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:12,248 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 02:54:12,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-03 02:54:12,249 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 02:54:12,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 02:54:12,250 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247 2023-04-03 02:54:12,251 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247 empty. 2023-04-03 02:54:12,251 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247 2023-04-03 02:54:12,251 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:54:12,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 02:54:12,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 02:54:12,661 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 02:54:12,662 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 03f67b48473b9761ac7a543789808247, NAME => 'TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp 2023-04-03 02:54:12,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 02:54:13,068 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:54:13,068 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 03f67b48473b9761ac7a543789808247, disabling compactions & flushes 2023-04-03 02:54:13,068 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:13,069 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:13,069 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. after waiting 0 ms 2023-04-03 02:54:13,069 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:13,069 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:13,069 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:13,071 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 02:54:13,072 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490453072"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490453072"}]},"ts":"1680490453072"} 2023-04-03 02:54:13,074 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 02:54:13,075 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 02:54:13,075 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490453075"}]},"ts":"1680490453075"} 2023-04-03 02:54:13,076 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 02:54:13,079 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03f67b48473b9761ac7a543789808247, ASSIGN}] 2023-04-03 02:54:13,080 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03f67b48473b9761ac7a543789808247, ASSIGN 2023-04-03 02:54:13,081 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=03f67b48473b9761ac7a543789808247, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43815,1680490421348; forceNewPlan=false, retain=false 2023-04-03 02:54:13,232 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=03f67b48473b9761ac7a543789808247, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:13,232 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490453232"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490453232"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490453232"}]},"ts":"1680490453232"} 2023-04-03 02:54:13,234 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:54:13,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 02:54:13,390 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:13,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 03f67b48473b9761ac7a543789808247, NAME => 'TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:54:13,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:13,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:54:13,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:13,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:13,391 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:13,393 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A 2023-04-03 02:54:13,393 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A 2023-04-03 02:54:13,393 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:54:13,393 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03f67b48473b9761ac7a543789808247 columnFamilyName A 2023-04-03 02:54:13,394 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(310): Store=03f67b48473b9761ac7a543789808247/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:54:13,394 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:13,395 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B 2023-04-03 02:54:13,395 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B 2023-04-03 02:54:13,395 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:54:13,395 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03f67b48473b9761ac7a543789808247 columnFamilyName B 2023-04-03 02:54:13,396 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(310): Store=03f67b48473b9761ac7a543789808247/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:54:13,396 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:13,397 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C 2023-04-03 02:54:13,397 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C 2023-04-03 02:54:13,397 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:54:13,397 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03f67b48473b9761ac7a543789808247 columnFamilyName C 2023-04-03 02:54:13,398 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(310): Store=03f67b48473b9761ac7a543789808247/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:54:13,398 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:13,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247 2023-04-03 02:54:13,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247 2023-04-03 02:54:13,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 02:54:13,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:13,403 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 02:54:13,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 03f67b48473b9761ac7a543789808247; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60067656, jitterRate=-0.10492217540740967}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 02:54:13,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:13,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., pid=18, masterSystemTime=1680490453385 2023-04-03 02:54:13,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:13,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:13,406 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=03f67b48473b9761ac7a543789808247, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:13,407 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490453406"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680490453406"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680490453406"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680490453406"}]},"ts":"1680490453406"} 2023-04-03 02:54:13,410 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-03 02:54:13,410 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 in 174 msec 2023-04-03 02:54:13,412 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-03 02:54:13,412 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=03f67b48473b9761ac7a543789808247, ASSIGN in 331 msec 2023-04-03 02:54:13,412 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 02:54:13,413 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490453412"}]},"ts":"1680490453412"} 2023-04-03 02:54:13,414 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 02:54:13,417 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 02:54:13,418 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1720 sec 2023-04-03 02:54:14,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 02:54:14,354 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-03 02:54:14,355 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x7829b43e to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:14,360 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@186ae08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:14,361 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:14,365 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:14,366 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 02:54:14,369 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55498, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 02:54:14,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 02:54:14,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 02:54:14,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:14,807 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 02:54:14,808 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 02:54:14,808 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 02:54:14,813 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-03 02:54:14,824 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03f67b48473b9761ac7a543789808247, REOPEN/MOVE}] 2023-04-03 02:54:14,825 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03f67b48473b9761ac7a543789808247, REOPEN/MOVE 2023-04-03 02:54:14,826 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=03f67b48473b9761ac7a543789808247, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:14,826 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490454826"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490454826"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490454826"}]},"ts":"1680490454826"} 2023-04-03 02:54:14,827 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:54:14,979 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:14,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 03f67b48473b9761ac7a543789808247, disabling compactions & flushes 2023-04-03 02:54:14,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:14,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:14,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. after waiting 0 ms 2023-04-03 02:54:14,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:14,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-03 02:54:14,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:14,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:14,984 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 03f67b48473b9761ac7a543789808247 to self. 2023-04-03 02:54:14,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:14,986 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=03f67b48473b9761ac7a543789808247, regionState=CLOSED 2023-04-03 02:54:14,987 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490454986"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490454986"}]},"ts":"1680490454986"} 2023-04-03 02:54:14,989 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-03 02:54:14,989 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 in 161 msec 2023-04-03 02:54:14,990 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=03f67b48473b9761ac7a543789808247, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,43815,1680490421348; forceNewPlan=false, retain=true 2023-04-03 02:54:15,140 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=03f67b48473b9761ac7a543789808247, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,141 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490455140"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490455140"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490455140"}]},"ts":"1680490455140"} 2023-04-03 02:54:15,142 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:54:15,297 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:15,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 03f67b48473b9761ac7a543789808247, NAME => 'TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:54:15,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:15,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:54:15,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:15,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:15,300 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:15,301 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A 2023-04-03 02:54:15,301 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A 2023-04-03 02:54:15,301 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:54:15,304 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03f67b48473b9761ac7a543789808247 columnFamilyName A 2023-04-03 02:54:15,306 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(310): Store=03f67b48473b9761ac7a543789808247/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:54:15,306 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:15,307 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B 2023-04-03 02:54:15,307 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B 2023-04-03 02:54:15,307 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:54:15,308 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03f67b48473b9761ac7a543789808247 columnFamilyName B 2023-04-03 02:54:15,308 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(310): Store=03f67b48473b9761ac7a543789808247/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:54:15,308 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:15,309 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C 2023-04-03 02:54:15,309 DEBUG [StoreOpener-03f67b48473b9761ac7a543789808247-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C 2023-04-03 02:54:15,309 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:54:15,310 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03f67b48473b9761ac7a543789808247 columnFamilyName C 2023-04-03 02:54:15,310 INFO [StoreOpener-03f67b48473b9761ac7a543789808247-1] regionserver.HStore(310): Store=03f67b48473b9761ac7a543789808247/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:54:15,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:15,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247 2023-04-03 02:54:15,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247 2023-04-03 02:54:15,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 02:54:15,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:15,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 03f67b48473b9761ac7a543789808247; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60729245, jitterRate=-0.09506373107433319}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 02:54:15,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:15,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., pid=23, masterSystemTime=1680490455294 2023-04-03 02:54:15,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:15,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:15,318 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=03f67b48473b9761ac7a543789808247, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,318 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490455318"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680490455318"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680490455318"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680490455318"}]},"ts":"1680490455318"} 2023-04-03 02:54:15,321 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-03 02:54:15,321 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 in 178 msec 2023-04-03 02:54:15,323 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-03 02:54:15,323 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=03f67b48473b9761ac7a543789808247, REOPEN/MOVE in 497 msec 2023-04-03 02:54:15,325 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-03 02:54:15,325 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 513 msec 2023-04-03 02:54:15,327 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 946 msec 2023-04-03 02:54:15,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-03 02:54:15,336 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x21281a37 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:15,341 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55cbb115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:15,342 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:15,346 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:15,346 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:15,350 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:15,351 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:15,358 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:15,358 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:15,364 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:15,364 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:15,369 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:15,369 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:15,373 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:15,374 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:15,377 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:15,377 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x2efdb02f to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:15,380 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@92bed09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:15,385 DEBUG [hconnection-0xfe637b3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:15,386 DEBUG [hconnection-0xf53a7db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:15,387 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:15,388 DEBUG [hconnection-0x748fa977-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:15,388 DEBUG [hconnection-0x4015b707-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:15,389 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60378, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:15,390 DEBUG [hconnection-0x77ff4f11-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:15,390 DEBUG [hconnection-0x556b156f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:15,393 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:15,394 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60362, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:15,394 DEBUG [hconnection-0x2d238d7d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:15,395 DEBUG [hconnection-0x63edd3be-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:15,396 DEBUG [hconnection-0x3df2f3e7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:15,397 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60390, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:15,397 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:15,401 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60380, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:15,401 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60402, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:15,401 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:15,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:15,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:15,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:15,415 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:15,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:15,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:15,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:15,416 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,416 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,417 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:15,417 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:15,417 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,417 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:15,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:15,422 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:15,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:15,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:54:15,423 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:15,423 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:15,423 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:15,423 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:15,423 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:15,423 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:15,425 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:15,425 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:15,425 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:15,425 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:15,425 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:15,425 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:15,425 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:15,425 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:15,425 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,425 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:15,426 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:15,436 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:15,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:15,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:15,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2966b6fd[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:15,439 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:15,440 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,441 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:15,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:15,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:15,441 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,441 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:15,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:15,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:15,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:15,442 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:15,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:15,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:15,447 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:15,447 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:15,447 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:15,447 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:15,447 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:15,449 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,449 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:15,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,449 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:15,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,450 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,452 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:15,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,460 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:15,460 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:15,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:15,460 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:15,461 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:15,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:15,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:15,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:15,461 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,461 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,461 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,461 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,461 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,461 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,461 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:15,461 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:15,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:15,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490515463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490515464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490515465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490515472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490515474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,477 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304032a7ee6d0d3e24272933c2e3c5775d1e5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032a7ee6d0d3e24272933c2e3c5775d1e5 2023-04-03 02:54:15,540 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/6f6c37a4e35142bf9536a6f7ddf6b918 2023-04-03 02:54:15,561 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:15,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:15,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:15,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:15,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:15,567 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:15,567 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:15,568 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:15,568 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:15,569 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,569 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490515570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,571 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:15,571 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:15,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:15,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,571 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,571 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:15,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490515570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:15,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:15,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:15,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:15,573 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:15,573 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:15,573 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:15,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490515571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490515577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490515579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:15,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:15,584 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:15,584 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:15,584 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:15,584 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:15,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:15,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:15,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,604 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:15,604 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:15,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,607 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:15,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:15,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:15,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc6d97b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:15,610 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:15,610 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,612 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,612 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:15,612 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:15,612 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,612 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:15,612 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:15,614 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:15,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:15,618 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:15,618 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:15,618 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:15,618 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:15,618 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:15,620 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,620 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:15,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:15,620 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:15,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:15,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,627 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:15,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:15,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:15,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:15,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:15,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:15,627 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:15,627 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:15,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:15,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:15,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:15,634 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/42991602ada2462995f145961aacfd74 2023-04-03 02:54:15,658 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/481c0c8f29744f1f9886a3da360c22c4 2023-04-03 02:54:15,666 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/6f6c37a4e35142bf9536a6f7ddf6b918 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6f6c37a4e35142bf9536a6f7ddf6b918 2023-04-03 02:54:15,672 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6f6c37a4e35142bf9536a6f7ddf6b918, entries=150, sequenceid=16, filesize=25.1 K 2023-04-03 02:54:15,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/42991602ada2462995f145961aacfd74 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/42991602ada2462995f145961aacfd74 2023-04-03 02:54:15,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/42991602ada2462995f145961aacfd74, entries=150, sequenceid=16, filesize=11.6 K 2023-04-03 02:54:15,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/481c0c8f29744f1f9886a3da360c22c4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/481c0c8f29744f1f9886a3da360c22c4 2023-04-03 02:54:15,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/481c0c8f29744f1f9886a3da360c22c4, entries=150, sequenceid=16, filesize=11.6 K 2023-04-03 02:54:15,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 03f67b48473b9761ac7a543789808247 in 272ms, sequenceid=16, compaction requested=false 2023-04-03 02:54:15,695 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-03 02:54:15,696 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:15,727 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:15,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:15,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:15,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:15,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:15,744 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:15,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:15,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:15,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:15,745 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,745 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,747 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:15,747 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:15,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:15,748 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,748 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:15,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:15,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:15,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:15,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:15,752 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:15,752 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:15,752 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:15,760 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:15,760 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:15,760 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:15,760 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:15,760 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 02:54:15,761 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:15,761 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:15,761 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:15,761 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:15,761 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:15,761 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:15,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:15,806 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:15,825 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403565bfd44a001439680e1e23a48931c97 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403565bfd44a001439680e1e23a48931c97 2023-04-03 02:54:15,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490515823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490515822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490515824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490515824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490515825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490515927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490515927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490515927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490515928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:15,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:15,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490515931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,003 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 02:54:16,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490516129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490516130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490516131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490516131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490516134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,250 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/a9a2366a90bb4dc0b77c43410fed531f 2023-04-03 02:54:16,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490516435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490516435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490516436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490516438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490516438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,668 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/2ad9030320214a3e9957599a8ff1fb17 2023-04-03 02:54:16,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490516938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490516940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490516941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490516942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:16,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:16,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490516943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,088 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/48ae617802254fe398cff0cbf9e25f86 2023-04-03 02:54:17,093 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/a9a2366a90bb4dc0b77c43410fed531f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a9a2366a90bb4dc0b77c43410fed531f 2023-04-03 02:54:17,098 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a9a2366a90bb4dc0b77c43410fed531f, entries=100, sequenceid=36, filesize=18.4 K 2023-04-03 02:54:17,098 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/2ad9030320214a3e9957599a8ff1fb17 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2ad9030320214a3e9957599a8ff1fb17 2023-04-03 02:54:17,107 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2ad9030320214a3e9957599a8ff1fb17, entries=100, sequenceid=36, filesize=9.3 K 2023-04-03 02:54:17,111 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/48ae617802254fe398cff0cbf9e25f86 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/48ae617802254fe398cff0cbf9e25f86 2023-04-03 02:54:17,117 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/48ae617802254fe398cff0cbf9e25f86, entries=100, sequenceid=36, filesize=9.3 K 2023-04-03 02:54:17,118 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 03f67b48473b9761ac7a543789808247 in 1358ms, sequenceid=36, compaction requested=false 2023-04-03 02:54:17,118 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:17,118 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:17,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:17,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:17,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:17,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:17,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:17,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:17,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:17,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:17,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:17,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:17,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:17,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@473129c3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:17,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:17,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,125 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,125 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:17,125 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:17,125 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:17,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:17,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:17,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:17,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:17,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:17,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:17,131 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:17,131 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:17,131 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:17,131 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:17,131 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:17,133 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,133 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:17,134 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:17,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:17,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,136 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,139 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:17,139 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,139 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,139 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,139 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:17,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:17,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:17,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:17,139 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:17,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:17,139 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,140 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,140 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,140 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:17,140 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:17,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:17,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:17,240 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:17,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:17,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:17,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:17,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:17,248 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:17,248 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:17,248 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:17,248 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:17,249 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,249 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,251 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,251 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:17,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:17,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:17,252 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,252 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:17,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:17,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:17,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:17,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:17,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:17,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:17,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:17,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:17,263 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:17,263 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:17,263 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 02:54:17,264 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:17,264 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:17,264 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:17,264 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:17,264 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:17,264 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:17,300 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304038731403200f44fd69c2e2ffa9ae6552c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038731403200f44fd69c2e2ffa9ae6552c 2023-04-03 02:54:17,311 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/908aea553c8a468786e07e8c0c30b5cc 2023-04-03 02:54:17,330 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/a54785d42c784dd29745e88e5578f53a 2023-04-03 02:54:17,756 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/75d699a260734ae7a35f8595436a9776 2023-04-03 02:54:17,763 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/908aea553c8a468786e07e8c0c30b5cc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/908aea553c8a468786e07e8c0c30b5cc 2023-04-03 02:54:17,769 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/908aea553c8a468786e07e8c0c30b5cc, entries=150, sequenceid=49, filesize=25.1 K 2023-04-03 02:54:17,770 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/a54785d42c784dd29745e88e5578f53a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/a54785d42c784dd29745e88e5578f53a 2023-04-03 02:54:17,776 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/a54785d42c784dd29745e88e5578f53a, entries=150, sequenceid=49, filesize=11.6 K 2023-04-03 02:54:17,777 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/75d699a260734ae7a35f8595436a9776 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/75d699a260734ae7a35f8595436a9776 2023-04-03 02:54:17,784 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/75d699a260734ae7a35f8595436a9776, entries=150, sequenceid=49, filesize=11.6 K 2023-04-03 02:54:17,787 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 03f67b48473b9761ac7a543789808247 in 524ms, sequenceid=49, compaction requested=true 2023-04-03 02:54:17,787 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:17,787 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:17,787 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:17,787 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:17,787 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:17,789 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:17,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:17,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ea63976[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:17,792 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:17,792 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,794 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:17,794 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:17,794 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:17,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:17,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:17,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:17,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:17,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:17,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:17,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:17,802 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:17,802 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:17,802 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:17,802 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:17,802 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:17,804 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,804 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:17,804 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,805 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:17,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,806 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,810 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:17,810 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:17,810 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:17,810 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:17,810 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:17,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:17,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:17,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,811 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:17,811 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:17,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:17,911 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:17,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:17,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:17,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:17,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:17,918 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:17,918 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:17,918 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:17,918 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:17,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,921 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:17,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:17,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:17,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:17,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:17,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:17,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:17,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:17,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:17,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:17,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:17,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:17,923 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:17,923 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:17,923 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:17,923 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:17,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:17,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:17,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:17,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:17,925 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:17,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:17,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:17,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62eba901[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:17,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:17,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,929 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:17,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:17,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:17,930 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,930 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:17,932 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:17,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:17,935 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:17,936 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:17,936 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:17,936 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:17,936 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:17,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,937 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,938 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:17,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,939 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:17,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:17,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:17,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:17,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:17,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:17,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:17,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:17,942 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:17,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:17,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:17,943 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:17,943 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:17,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:17,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:17,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:54:17,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:17,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:17,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:17,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:17,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:17,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:18,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490518016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490518018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490458000, max timestamp=1680490458021Num uniques 859; 2023-04-03 02:54:18,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490518019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490458000, max timestamp=1680490458021Num uniques 859; 2023-04-03 02:54:18,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:54:18,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=509, cellSize=23305, totalHeapSize=53176, min timestamp=1680490458000, max timestamp=1680490458021Num uniques 859; 2023-04-03 02:54:18,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:54:18,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=23305, suffix heap size=117064,new segment heap size=53176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=509 2023-04-03 02:54:18,039 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403f631696fbf144e9e8e70f7063d617e4f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f631696fbf144e9e8e70f7063d617e4f 2023-04-03 02:54:18,043 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:18,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:18,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:18,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:18,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:18,050 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:18,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:18,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:18,051 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,051 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,052 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,052 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,052 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,053 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:18,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:18,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:18,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:18,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:18,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490518050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490518050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:18,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:18,056 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,056 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:18,056 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:18,056 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:18,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:18,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:18,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:18,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,059 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:18,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292da926[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:18,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:18,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:18,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:18,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:18,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:18,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:18,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:18,068 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:18,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:18,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:18,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:18,068 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:18,069 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,069 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,070 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,071 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,072 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,076 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:18,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:18,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,076 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,076 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:18,076 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:18,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:18,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490518124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490518125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490518125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490518156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490518156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,176 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:18,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:18,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:18,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:18,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:18,180 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:18,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:18,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:18,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,182 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,182 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,182 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,182 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:18,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:18,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:18,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:18,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:18,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:18,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:18,185 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,185 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:18,185 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:18,185 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:18,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:18,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:18,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:18,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:18,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@671cf2f1[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:18,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:18,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,191 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,191 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:18,191 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:18,191 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:18,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:18,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,193 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,193 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,194 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:18,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:18,197 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:18,197 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:18,197 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:18,197 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:18,197 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:18,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,201 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,202 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,207 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:18,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:18,208 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:18,208 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:18,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,308 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:18,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:18,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:18,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:18,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:18,312 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:18,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:18,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:18,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,315 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,315 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,315 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,315 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:18,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:18,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:18,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:18,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:18,320 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,320 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:18,320 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:18,320 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:18,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:18,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:18,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:18,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:18,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:18,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28be0aed[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:18,324 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:18,324 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,325 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,325 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:18,325 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:18,325 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:18,325 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,326 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:18,327 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:18,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:18,330 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:18,330 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:18,330 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:18,330 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:18,330 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:18,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490518328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490518327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490518328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,335 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,336 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,345 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:18,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:18,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,345 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,345 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:18,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490518362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490518362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,445 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:18,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:18,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:18,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:18,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:18,449 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:18,449 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,449 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:18,449 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:18,450 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,450 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,451 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/1270d9a88a404304872bf863b0a9ba21 2023-04-03 02:54:18,451 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,451 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,452 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,452 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:18,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:18,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:18,455 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,455 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:18,455 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:18,455 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:18,457 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,457 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,457 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,457 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:18,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17220d5a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:18,459 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:18,460 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,461 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,462 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:18,462 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:18,462 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:18,462 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,462 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:18,463 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:18,463 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,463 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,463 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:18,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:18,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:18,468 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:18,468 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:18,468 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:18,468 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:18,468 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:18,469 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,469 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,469 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,472 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,474 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,476 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,476 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,476 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,476 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,476 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:18,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:18,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,476 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,476 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,476 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,477 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:18,477 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:18,476 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,477 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,577 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:18,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:18,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:18,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:18,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:18,580 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:18,580 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,580 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:18,580 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:18,581 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,581 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,582 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,582 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,582 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,582 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:18,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:18,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:18,583 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:18,583 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:18,584 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,584 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:18,584 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:18,584 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:18,585 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,585 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,586 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,586 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:18,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5382f201[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:18,587 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:18,587 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,588 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,588 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:18,588 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:18,588 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:18,588 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,588 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:18,596 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,596 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,596 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,596 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:18,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:18,598 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:18,598 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:18,598 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:18,598 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:18,598 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:18,600 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,600 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,600 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,603 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,606 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,606 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,606 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,606 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,606 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:18,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:18,606 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,606 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,606 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,606 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,606 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:18,606 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:18,607 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490518632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490518633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490518634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490518665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:18,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490518666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,706 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:18,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:18,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:18,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:18,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:18,710 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:18,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:18,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:18,711 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,711 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,712 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,712 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,713 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:18,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:18,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:18,714 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,714 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:18,714 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:18,714 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:18,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:18,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,716 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,716 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,716 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:18,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@530919fa[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:18,718 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:18,718 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,719 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,719 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:18,719 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:18,719 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,719 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:18,719 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:18,721 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,721 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:18,721 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,721 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:18,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:18,724 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:18,724 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:18,724 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:18,724 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:18,724 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:18,726 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,726 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,726 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,726 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,728 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,730 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,730 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,730 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,730 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,730 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:18,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:18,730 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,730 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,730 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,730 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:18,731 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,731 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:18,731 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,831 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:18,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:18,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:18,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:18,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:18,834 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:18,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:18,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:18,835 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,835 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,836 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,836 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,836 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,837 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:18,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:18,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:18,838 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:18,838 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,838 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:18,840 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,840 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:18,840 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:18,840 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,841 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:18,841 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:18,843 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,843 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,843 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,843 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:18,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b388814[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:18,845 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:18,845 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,846 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,846 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:18,846 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:18,846 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:18,846 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,846 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:18,847 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:18,847 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,847 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,847 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:18,850 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:18,850 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:18,850 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:18,850 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:18,850 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:18,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,852 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,853 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,856 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,856 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,856 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:18,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:18,856 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,856 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,857 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,857 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,857 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,857 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,856 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,857 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:18,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:18,857 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:18,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/56ba68408e66421e80ff7a52dfa1dd17 2023-04-03 02:54:18,957 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:18,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:18,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:18,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:18,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:18,961 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:18,961 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,961 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:18,961 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:18,961 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,961 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,962 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,962 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,963 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,963 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:18,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:18,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:18,964 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,964 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:18,964 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:18,964 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:18,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c771cf4[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:18,968 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:18,968 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,969 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:18,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:18,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:18,969 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,969 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:18,972 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:18,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:18,974 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:18,974 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:18,974 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:18,974 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:18,974 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:18,976 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,976 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,976 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:18,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:18,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,976 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,978 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:18,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:18,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:18,982 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:18,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:18,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:18,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:18,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:18,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:18,983 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:18,983 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:18,983 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:18,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:18,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:19,083 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:19,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:19,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:19,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:19,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:19,088 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:19,088 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:19,088 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:19,088 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:19,089 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,089 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,090 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,090 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:19,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:19,091 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,091 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:19,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:19,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:19,100 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:19,100 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:19,100 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:19,100 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:19,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,101 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:19,101 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:19,102 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48a1f581[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:19,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:19,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,112 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:19,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:19,114 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,114 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:19,116 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:19,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:19,120 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:19,120 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:19,120 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:19,120 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:19,120 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:19,122 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,122 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:19,122 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:19,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:19,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:19,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:19,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490519137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:19,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490519137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,141 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,142 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:19,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:19,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:19,142 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:19,142 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:19,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:19,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:19,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:19,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:19,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:19,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490519140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:19,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:19,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:19,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490519170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:19,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490519174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,242 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:19,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:19,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:19,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:19,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:19,245 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:19,246 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:19,246 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:19,246 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:19,246 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,246 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:19,247 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:19,248 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,248 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:19,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:19,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:19,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:19,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:19,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:19,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:19,250 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:19,250 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:19,250 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:19,250 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:19,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:19,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:19,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:19,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:19,253 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,253 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:19,254 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,254 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:19,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:19,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53fce3a6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:19,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:19,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,258 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:19,259 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,259 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:19,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:19,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:19,260 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:19,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:19,267 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:19,267 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:19,267 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:19,267 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:19,267 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:19,270 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:19,270 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,271 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:19,272 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,272 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,274 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,274 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:19,274 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,274 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:19,274 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:19,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:19,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:19,274 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:19,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:19,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:19,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:19,275 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:19,275 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:19,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:19,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:19,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/76894e1b1ef8452ea6002595bbd23dd7 2023-04-03 02:54:19,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/1270d9a88a404304872bf863b0a9ba21 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/1270d9a88a404304872bf863b0a9ba21 2023-04-03 02:54:19,303 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/1270d9a88a404304872bf863b0a9ba21, entries=150, sequenceid=61, filesize=25.1 K 2023-04-03 02:54:19,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/56ba68408e66421e80ff7a52dfa1dd17 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/56ba68408e66421e80ff7a52dfa1dd17 2023-04-03 02:54:19,310 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/56ba68408e66421e80ff7a52dfa1dd17, entries=150, sequenceid=61, filesize=11.6 K 2023-04-03 02:54:19,311 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/76894e1b1ef8452ea6002595bbd23dd7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/76894e1b1ef8452ea6002595bbd23dd7 2023-04-03 02:54:19,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/76894e1b1ef8452ea6002595bbd23dd7, entries=150, sequenceid=61, filesize=11.6 K 2023-04-03 02:54:19,317 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 03f67b48473b9761ac7a543789808247 in 1323ms, sequenceid=61, compaction requested=true 2023-04-03 02:54:19,317 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:19,317 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:19,317 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:54:19,317 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:19,317 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:54:19,317 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:19,319 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:54:19,319 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 96036 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:54:19,319 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/B is initiating minor compaction (all files) 2023-04-03 02:54:19,319 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/A is initiating minor compaction (all files) 2023-04-03 02:54:19,319 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/B in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:19,319 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/A in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:19,319 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/42991602ada2462995f145961aacfd74, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2ad9030320214a3e9957599a8ff1fb17, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/a54785d42c784dd29745e88e5578f53a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/56ba68408e66421e80ff7a52dfa1dd17] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=44.1 K 2023-04-03 02:54:19,319 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6f6c37a4e35142bf9536a6f7ddf6b918, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a9a2366a90bb4dc0b77c43410fed531f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/908aea553c8a468786e07e8c0c30b5cc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/1270d9a88a404304872bf863b0a9ba21] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=93.8 K 2023-04-03 02:54:19,320 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 42991602ada2462995f145961aacfd74, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680490455413 2023-04-03 02:54:19,320 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6f6c37a4e35142bf9536a6f7ddf6b918, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680490455413 2023-04-03 02:54:19,320 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 2ad9030320214a3e9957599a8ff1fb17, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680490455462 2023-04-03 02:54:19,320 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a9a2366a90bb4dc0b77c43410fed531f, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680490455462 2023-04-03 02:54:19,320 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a54785d42c784dd29745e88e5578f53a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680490455821 2023-04-03 02:54:19,321 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 908aea553c8a468786e07e8c0c30b5cc, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680490455821 2023-04-03 02:54:19,321 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 56ba68408e66421e80ff7a52dfa1dd17, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680490457988 2023-04-03 02:54:19,321 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 1270d9a88a404304872bf863b0a9ba21, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680490457988 2023-04-03 02:54:19,332 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#B#compaction#126 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:19,338 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#A#compaction#127 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:19,346 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/cc5b7176632f40f2b0e2c402fb9d6fce as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/cc5b7176632f40f2b0e2c402fb9d6fce 2023-04-03 02:54:19,352 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 03f67b48473b9761ac7a543789808247/B of 03f67b48473b9761ac7a543789808247 into cc5b7176632f40f2b0e2c402fb9d6fce(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:19,352 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:19,352 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/B, priority=12, startTime=1680490459317; duration=0sec 2023-04-03 02:54:19,353 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:19,353 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:54:19,354 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:54:19,354 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/C is initiating minor compaction (all files) 2023-04-03 02:54:19,355 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/C in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:19,355 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/481c0c8f29744f1f9886a3da360c22c4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/48ae617802254fe398cff0cbf9e25f86, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/75d699a260734ae7a35f8595436a9776, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/76894e1b1ef8452ea6002595bbd23dd7] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=44.1 K 2023-04-03 02:54:19,356 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 481c0c8f29744f1f9886a3da360c22c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680490455413 2023-04-03 02:54:19,357 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 48ae617802254fe398cff0cbf9e25f86, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680490455462 2023-04-03 02:54:19,357 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 75d699a260734ae7a35f8595436a9776, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680490455821 2023-04-03 02:54:19,358 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 76894e1b1ef8452ea6002595bbd23dd7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680490457988 2023-04-03 02:54:19,375 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:19,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:19,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:19,382 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#C#compaction#128 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:19,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:19,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:19,396 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:19,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:19,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:19,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:19,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,398 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:19,398 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:19,399 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:19,399 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:19,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:19,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:19,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:19,404 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:19,404 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:19,404 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-03 02:54:19,405 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:19,405 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:19,405 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:19,405 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:19,405 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:19,405 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:19,824 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/60ab3823eb1d4dddb509dd6fcafdd047 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/60ab3823eb1d4dddb509dd6fcafdd047 2023-04-03 02:54:19,830 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/6df0ef07060344f8b45cb72521211123 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6df0ef07060344f8b45cb72521211123 2023-04-03 02:54:19,833 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 03f67b48473b9761ac7a543789808247/C of 03f67b48473b9761ac7a543789808247 into 60ab3823eb1d4dddb509dd6fcafdd047(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:19,833 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:19,833 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/C, priority=12, startTime=1680490459317; duration=0sec 2023-04-03 02:54:19,833 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:19,836 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304030bf9938d731c4379bf5d7d693041d037 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030bf9938d731c4379bf5d7d693041d037 2023-04-03 02:54:19,845 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 03f67b48473b9761ac7a543789808247/A of 03f67b48473b9761ac7a543789808247 into 6df0ef07060344f8b45cb72521211123(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:19,845 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:19,845 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/A, priority=12, startTime=1680490459317; duration=0sec 2023-04-03 02:54:19,845 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:19,858 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=96, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/366f6d0c1c544dadbf65f0f30bd3d4a0 2023-04-03 02:54:20,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:20,143 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:20,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490520150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490520152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490520153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490520174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490520180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490520254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490520254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490520256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,297 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/0eed3c3cc7534e79b8535bd9df3653e0 2023-04-03 02:54:20,309 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/d0eef38a953d44749e3c1e993db33406 2023-04-03 02:54:20,314 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/366f6d0c1c544dadbf65f0f30bd3d4a0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/366f6d0c1c544dadbf65f0f30bd3d4a0 2023-04-03 02:54:20,319 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/366f6d0c1c544dadbf65f0f30bd3d4a0, entries=150, sequenceid=96, filesize=25.1 K 2023-04-03 02:54:20,320 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/0eed3c3cc7534e79b8535bd9df3653e0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0eed3c3cc7534e79b8535bd9df3653e0 2023-04-03 02:54:20,328 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0eed3c3cc7534e79b8535bd9df3653e0, entries=150, sequenceid=96, filesize=11.6 K 2023-04-03 02:54:20,329 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/d0eef38a953d44749e3c1e993db33406 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d0eef38a953d44749e3c1e993db33406 2023-04-03 02:54:20,335 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d0eef38a953d44749e3c1e993db33406, entries=150, sequenceid=96, filesize=11.6 K 2023-04-03 02:54:20,335 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 03f67b48473b9761ac7a543789808247 in 931ms, sequenceid=96, compaction requested=false 2023-04-03 02:54:20,336 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:20,336 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:20,336 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:20,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c4bb9e5[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:20,339 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:20,339 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,342 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,342 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:20,342 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:20,342 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:20,342 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,342 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:20,343 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:20,346 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:20,346 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:20,346 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:20,346 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:20,346 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:20,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,348 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,348 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:20,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,348 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,349 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,353 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:20,353 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,353 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:20,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:20,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:20,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:20,353 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:20,353 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,353 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:20,353 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:20,353 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:20,353 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,353 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,353 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,354 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,453 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:20,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:20,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:20,457 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:20,457 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:20,457 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:20,458 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:20,458 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,458 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:20,459 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:54:20,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:20,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:20,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:20,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:20,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:20,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:20,465 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:20,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:20,465 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,466 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,466 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:20,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:20,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:20,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:20,468 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:20,468 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:20,468 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:20,468 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:20,468 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:20,468 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:20,468 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:20,468 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:20,468 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:20,468 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:20,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:20,475 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,475 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a3187788a622416f890c6e5eb56048c2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a3187788a622416f890c6e5eb56048c2 2023-04-03 02:54:20,476 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,476 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@286f8624[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:20,478 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:20,478 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,479 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:20,480 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,480 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:20,481 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:20,481 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:20,481 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,481 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:20,484 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:20,484 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:20,484 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:20,484 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:20,484 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:20,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490520482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,487 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,487 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:20,487 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,489 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490520486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490520487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,491 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/21a5e10e6e6346db906531c1da975575 2023-04-03 02:54:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,496 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:20,496 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:20,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:20,496 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,496 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,496 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,496 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,496 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,496 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,496 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,496 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:20,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:20,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:20,497 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:20,497 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:20,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:20,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490520588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490520591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490520591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,597 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:20,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:20,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:20,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:20,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:20,601 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:20,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:20,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:20,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:20,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,605 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,605 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:20,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:20,605 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,605 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:20,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:20,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:20,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:20,607 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:20,607 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:20,608 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:20,608 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:20,609 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:20,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa57355[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:20,614 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:20,614 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,616 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,616 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:20,616 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:20,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:20,616 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:20,618 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:20,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:20,621 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:20,621 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:20,621 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:20,621 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:20,621 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:20,623 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,623 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,624 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,626 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,629 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:20,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:20,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:20,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:20,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:20,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:20,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,629 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:20,629 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:20,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:20,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:20,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,729 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:20,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:20,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:20,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:20,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:20,736 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:20,736 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:20,736 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:20,736 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:20,737 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,737 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,738 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,738 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:20,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,738 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,738 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:20,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:20,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:20,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:20,740 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:20,740 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:20,740 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:20,740 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:20,742 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,742 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:20,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bf05cd4[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:20,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:20,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,747 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:20,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:20,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:20,747 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,747 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:20,750 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:20,756 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:20,756 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:20,757 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:20,757 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:20,757 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:20,758 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,758 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,758 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:20,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,759 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,761 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,764 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:20,764 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:20,764 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,764 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:20,764 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:20,764 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:20,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:20,764 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,765 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,765 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,765 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,765 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:20,765 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:20,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:20,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490520790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490520793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:20,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490520793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,865 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:20,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:20,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:20,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:20,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:20,869 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:20,869 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:20,869 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:20,869 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:20,869 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,869 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,870 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:20,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:20,871 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,871 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:20,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:20,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:20,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:20,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:20,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:20,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:20,874 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:20,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab542de[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:20,876 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:20,876 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,877 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:20,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:20,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:20,878 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,878 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:20,881 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:20,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:20,885 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:20,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:20,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:20,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:20,886 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:20,887 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,887 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:20,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,887 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:20,888 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,889 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:20,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:20,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:20,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:20,892 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:20,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:20,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:20,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:20,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,892 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:20,892 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:20,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:20,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:20,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:20,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:20,937 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/0a6096f612e241a081f5d4ded287ad6a 2023-04-03 02:54:20,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/ff4aa26c11f54c8d96d2582c0d54ffad 2023-04-03 02:54:20,965 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/21a5e10e6e6346db906531c1da975575 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/21a5e10e6e6346db906531c1da975575 2023-04-03 02:54:20,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/21a5e10e6e6346db906531c1da975575, entries=200, sequenceid=110, filesize=31.9 K 2023-04-03 02:54:20,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/0a6096f612e241a081f5d4ded287ad6a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0a6096f612e241a081f5d4ded287ad6a 2023-04-03 02:54:20,976 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0a6096f612e241a081f5d4ded287ad6a, entries=150, sequenceid=110, filesize=11.6 K 2023-04-03 02:54:20,977 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/ff4aa26c11f54c8d96d2582c0d54ffad as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/ff4aa26c11f54c8d96d2582c0d54ffad 2023-04-03 02:54:20,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/ff4aa26c11f54c8d96d2582c0d54ffad, entries=150, sequenceid=110, filesize=11.6 K 2023-04-03 02:54:20,982 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 03f67b48473b9761ac7a543789808247 in 523ms, sequenceid=110, compaction requested=true 2023-04-03 02:54:20,983 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:20,983 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:20,983 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:20,983 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:20,983 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:20,983 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:20,984 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84316 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:20,984 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:20,984 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/A is initiating minor compaction (all files) 2023-04-03 02:54:20,984 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/B is initiating minor compaction (all files) 2023-04-03 02:54:20,984 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/A in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:20,984 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/B in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:20,984 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6df0ef07060344f8b45cb72521211123, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/366f6d0c1c544dadbf65f0f30bd3d4a0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/21a5e10e6e6346db906531c1da975575] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=82.3 K 2023-04-03 02:54:20,984 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/cc5b7176632f40f2b0e2c402fb9d6fce, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0eed3c3cc7534e79b8535bd9df3653e0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0a6096f612e241a081f5d4ded287ad6a] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=35.0 K 2023-04-03 02:54:20,985 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6df0ef07060344f8b45cb72521211123, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680490457988 2023-04-03 02:54:20,985 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting cc5b7176632f40f2b0e2c402fb9d6fce, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680490457988 2023-04-03 02:54:20,986 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 366f6d0c1c544dadbf65f0f30bd3d4a0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680490458032 2023-04-03 02:54:20,986 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 0eed3c3cc7534e79b8535bd9df3653e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680490458032 2023-04-03 02:54:20,986 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 21a5e10e6e6346db906531c1da975575, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680490460148 2023-04-03 02:54:20,986 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 0a6096f612e241a081f5d4ded287ad6a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680490460148 2023-04-03 02:54:20,992 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:20,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:20,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:21,001 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#B#compaction#135 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:21,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:21,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:21,002 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:21,002 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:21,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:21,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:21,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:21,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:21,006 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,006 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:21,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:21,006 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,006 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:21,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:21,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:21,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:21,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:21,010 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:21,017 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:21,017 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:21,018 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:21,018 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:21,018 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 02:54:21,018 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:21,018 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:21,018 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:21,018 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:21,018 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:21,018 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:21,028 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/8ddd6c89d3734f9c857465390761c6d2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/8ddd6c89d3734f9c857465390761c6d2 2023-04-03 02:54:21,033 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#A#compaction#136 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:21,036 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 03f67b48473b9761ac7a543789808247/B of 03f67b48473b9761ac7a543789808247 into 8ddd6c89d3734f9c857465390761c6d2(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:21,036 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:21,036 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/B, priority=13, startTime=1680490460983; duration=0sec 2023-04-03 02:54:21,036 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:21,036 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:21,039 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:21,040 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/C is initiating minor compaction (all files) 2023-04-03 02:54:21,040 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/C in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:21,040 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/60ab3823eb1d4dddb509dd6fcafdd047, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d0eef38a953d44749e3c1e993db33406, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/ff4aa26c11f54c8d96d2582c0d54ffad] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=35.0 K 2023-04-03 02:54:21,040 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 60ab3823eb1d4dddb509dd6fcafdd047, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680490457988 2023-04-03 02:54:21,041 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d0eef38a953d44749e3c1e993db33406, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680490458032 2023-04-03 02:54:21,041 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ff4aa26c11f54c8d96d2582c0d54ffad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680490460148 2023-04-03 02:54:21,055 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#C#compaction#138 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:21,078 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403b35a6953ff6540ff831d456a69cdee75 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b35a6953ff6540ff831d456a69cdee75 2023-04-03 02:54:21,097 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:21,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:21,100 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/27f16b30f1014082999238a33d3d2c4e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/27f16b30f1014082999238a33d3d2c4e 2023-04-03 02:54:21,105 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 03f67b48473b9761ac7a543789808247/C of 03f67b48473b9761ac7a543789808247 into 27f16b30f1014082999238a33d3d2c4e(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:21,106 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:21,106 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/C, priority=13, startTime=1680490460983; duration=0sec 2023-04-03 02:54:21,106 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:21,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490521113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490521119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490521128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490521221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490521225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490521229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490521423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490521428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490521432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,466 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/a0d8bceb1ca04bcdb9a5e5c9f57c78fc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a0d8bceb1ca04bcdb9a5e5c9f57c78fc 2023-04-03 02:54:21,471 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 03f67b48473b9761ac7a543789808247/A of 03f67b48473b9761ac7a543789808247 into a0d8bceb1ca04bcdb9a5e5c9f57c78fc(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:21,471 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:21,471 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/A, priority=13, startTime=1680490460983; duration=0sec 2023-04-03 02:54:21,471 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:21,515 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/0ac1d0d5e03a4aa6bf94a56ab46a72ce 2023-04-03 02:54:21,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490521725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490521733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490521734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,931 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/94cb3c513bce486d95f4066fd850c636 2023-04-03 02:54:21,942 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/0fc0716601424e70b88e27d06addf2b9 2023-04-03 02:54:21,947 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/0ac1d0d5e03a4aa6bf94a56ab46a72ce as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/0ac1d0d5e03a4aa6bf94a56ab46a72ce 2023-04-03 02:54:21,951 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/0ac1d0d5e03a4aa6bf94a56ab46a72ce, entries=150, sequenceid=130, filesize=25.2 K 2023-04-03 02:54:21,952 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/94cb3c513bce486d95f4066fd850c636 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/94cb3c513bce486d95f4066fd850c636 2023-04-03 02:54:21,966 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/94cb3c513bce486d95f4066fd850c636, entries=150, sequenceid=130, filesize=11.7 K 2023-04-03 02:54:21,967 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/0fc0716601424e70b88e27d06addf2b9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/0fc0716601424e70b88e27d06addf2b9 2023-04-03 02:54:21,973 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/0fc0716601424e70b88e27d06addf2b9, entries=150, sequenceid=130, filesize=11.7 K 2023-04-03 02:54:21,974 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 03f67b48473b9761ac7a543789808247 in 956ms, sequenceid=130, compaction requested=false 2023-04-03 02:54:21,974 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:21,974 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:21,974 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:21,975 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:21,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:21,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ebfa90a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:21,978 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:21,978 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:21,979 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:21,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:21,979 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:21,979 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:21,979 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:21,980 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,980 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:21,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:21,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:21,985 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:21,985 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:21,985 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:21,985 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:21,985 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:21,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:21,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:21,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:21,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:21,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:21,988 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,989 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,991 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,991 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:21,991 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:21,991 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:21,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:21,991 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:21,991 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:21,991 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:21,991 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:21,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:21,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:21,991 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:21,992 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:21,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:21,992 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:21,992 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:22,092 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:22,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:22,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:22,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:22,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:22,108 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:22,108 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:22,108 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:22,109 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:22,109 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:22,109 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:22,110 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:22,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:22,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:22,113 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,113 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:22,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:22,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:22,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:22,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:22,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:22,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:22,117 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:22,117 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:22,117 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 02:54:22,117 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:22,117 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:22,118 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:22,118 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:22,118 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:22,118 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:22,131 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304032a4f8c020fbd4308bb1432d6f5a5af2c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032a4f8c020fbd4308bb1432d6f5a5af2c 2023-04-03 02:54:22,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:22,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:22,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:22,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490522214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:22,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490522216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:22,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490522231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:22,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490522236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:22,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490522239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:22,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490522319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:22,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490522320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:22,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490522520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:22,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490522524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,541 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/93754b009f3b4a18b5a4cc532a116eb4 2023-04-03 02:54:22,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:22,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490522824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:22,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490522829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:22,960 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/f4b1cb10fcc9454aaa89cd31f48ba74b 2023-04-03 02:54:22,981 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/add2905a180248d28995981e6f34dad7 2023-04-03 02:54:22,986 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/93754b009f3b4a18b5a4cc532a116eb4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/93754b009f3b4a18b5a4cc532a116eb4 2023-04-03 02:54:22,990 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/93754b009f3b4a18b5a4cc532a116eb4, entries=150, sequenceid=146, filesize=25.3 K 2023-04-03 02:54:22,991 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/f4b1cb10fcc9454aaa89cd31f48ba74b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/f4b1cb10fcc9454aaa89cd31f48ba74b 2023-04-03 02:54:22,995 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/f4b1cb10fcc9454aaa89cd31f48ba74b, entries=150, sequenceid=146, filesize=11.8 K 2023-04-03 02:54:22,996 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/add2905a180248d28995981e6f34dad7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/add2905a180248d28995981e6f34dad7 2023-04-03 02:54:23,000 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/add2905a180248d28995981e6f34dad7, entries=150, sequenceid=146, filesize=11.8 K 2023-04-03 02:54:23,001 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 03f67b48473b9761ac7a543789808247 in 884ms, sequenceid=146, compaction requested=true 2023-04-03 02:54:23,001 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:23,002 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:23,003 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,003 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:23,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:23,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a445453[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:23,005 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:23,005 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:23,006 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:23,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:23,006 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:23,006 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:23,006 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:23,006 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,006 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:23,008 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:23,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:23,011 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:23,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:23,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:23,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:23,011 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:23,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:23,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:23,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:23,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:23,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:23,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:23,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:23,017 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:23,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:23,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:23,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:23,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:23,020 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:23,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:23,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:23,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:23,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:23,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:23,020 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:23,020 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:23,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:23,120 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:23,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:23,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:23,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:23,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:23,124 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:23,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:23,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:23,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:23,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:23,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:23,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,125 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:23,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:23,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:23,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:23,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:23,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:23,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:23,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:23,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:23,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:23,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:23,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:23,127 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:23,127 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:23,127 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 02:54:23,127 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:23,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:23,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:23,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:23,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:23,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:23,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:23,236 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:23,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490523254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490523255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490523255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490523329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490523333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490523356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490523358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490523358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,541 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a45269d1a32c4ce19cc27e30fdd5701d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a45269d1a32c4ce19cc27e30fdd5701d 2023-04-03 02:54:23,547 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/d1ccf912cca348e5b6276565b0c8925d 2023-04-03 02:54:23,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490523559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490523561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490523561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490523862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490523864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:23,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490523866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:23,959 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/62d77a9ac9b841b7b8529e06db9af5c2 2023-04-03 02:54:23,996 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/f810cc6ef021415dbe6df1f00dd47628 2023-04-03 02:54:24,003 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/d1ccf912cca348e5b6276565b0c8925d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d1ccf912cca348e5b6276565b0c8925d 2023-04-03 02:54:24,012 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d1ccf912cca348e5b6276565b0c8925d, entries=150, sequenceid=165, filesize=25.3 K 2023-04-03 02:54:24,014 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/62d77a9ac9b841b7b8529e06db9af5c2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/62d77a9ac9b841b7b8529e06db9af5c2 2023-04-03 02:54:24,019 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/62d77a9ac9b841b7b8529e06db9af5c2, entries=150, sequenceid=165, filesize=11.8 K 2023-04-03 02:54:24,020 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/f810cc6ef021415dbe6df1f00dd47628 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/f810cc6ef021415dbe6df1f00dd47628 2023-04-03 02:54:24,024 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/f810cc6ef021415dbe6df1f00dd47628, entries=150, sequenceid=165, filesize=11.8 K 2023-04-03 02:54:24,025 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 03f67b48473b9761ac7a543789808247 in 898ms, sequenceid=165, compaction requested=true 2023-04-03 02:54:24,026 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:24,026 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,026 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:24,026 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:24,026 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,028 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:24,028 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:24,031 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@538bbf33[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:24,033 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:24,033 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,035 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,035 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:24,035 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:24,035 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:24,035 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,035 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:24,036 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:24,037 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:24,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:24,040 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:24,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:24,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:24,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:24,040 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:24,049 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,049 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,050 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,051 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,061 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:24,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:24,061 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,061 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:24,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:24,061 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:24,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,161 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:24,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:24,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:24,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:24,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:24,165 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:24,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:24,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:24,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,167 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,168 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,168 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:24,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:24,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:24,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:24,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:24,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:24,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:24,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:24,169 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,169 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:24,170 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 02:54:24,170 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:24,170 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:24,170 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:24,170 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:24,170 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:24,170 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:24,198 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403810c3b4b7c3347dab09a6619b8a1f3b3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403810c3b4b7c3347dab09a6619b8a1f3b3 2023-04-03 02:54:24,210 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=179, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/3be4fed221cf49fea053da487e12a37a 2023-04-03 02:54:24,237 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/749f9735289f42b9b24cbe86bcae0298 2023-04-03 02:54:24,275 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/bfcc7e1c22c14b549262d3d2eadf3caa 2023-04-03 02:54:24,281 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/3be4fed221cf49fea053da487e12a37a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3be4fed221cf49fea053da487e12a37a 2023-04-03 02:54:24,289 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3be4fed221cf49fea053da487e12a37a, entries=150, sequenceid=179, filesize=25.3 K 2023-04-03 02:54:24,290 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/749f9735289f42b9b24cbe86bcae0298 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/749f9735289f42b9b24cbe86bcae0298 2023-04-03 02:54:24,296 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/749f9735289f42b9b24cbe86bcae0298, entries=150, sequenceid=179, filesize=11.8 K 2023-04-03 02:54:24,297 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/bfcc7e1c22c14b549262d3d2eadf3caa as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/bfcc7e1c22c14b549262d3d2eadf3caa 2023-04-03 02:54:24,315 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/bfcc7e1c22c14b549262d3d2eadf3caa, entries=150, sequenceid=179, filesize=11.8 K 2023-04-03 02:54:24,316 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 03f67b48473b9761ac7a543789808247 in 147ms, sequenceid=179, compaction requested=true 2023-04-03 02:54:24,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:24,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:24,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:24,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:24,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:24,318 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,319 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,319 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:24,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6318b42e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:24,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:24,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,323 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,323 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:24,330 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,330 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:24,330 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:24,330 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,332 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:24,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:24,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:24,335 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:24,335 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:24,336 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:24,336 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:24,336 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:24,339 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,339 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,339 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,340 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,341 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,344 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,344 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,344 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,344 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:24,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:24,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:24,344 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,344 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,344 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,344 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,344 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:24,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,344 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,344 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:24,344 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:24,387 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:54:24,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:24,387 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:24,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:24,387 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:24,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:24,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:24,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490464394, max timestamp=1680490464424Num uniques 859; 2023-04-03 02:54:24,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490464394, max timestamp=1680490464424Num uniques 859; 2023-04-03 02:54:24,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:54:24,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490464394, max timestamp=1680490464424Num uniques 859; 2023-04-03 02:54:24,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:54:24,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:54:24,445 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:24,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:24,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:24,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490524445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490524445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490524446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490524447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490524451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:24,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:24,456 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:24,456 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,456 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:24,456 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:24,456 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,457 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,458 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,458 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,458 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,458 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:24,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:24,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:24,461 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:24,461 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,461 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:24,464 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,464 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:24,464 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:24,464 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:24,466 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,466 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,466 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,466 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:24,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50dca6be[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:24,468 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:24,468 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,475 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,475 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:24,475 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:24,475 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:24,476 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,476 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:24,480 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:24,480 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,480 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,480 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:24,483 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:24,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:24,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:24,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:24,483 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:24,486 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,486 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,486 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,487 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,489 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,491 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,491 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,491 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,491 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,491 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:24,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,491 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,492 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:24,492 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:24,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490524555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490524555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490524555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490524556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490524556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,592 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:24,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:24,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:24,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:24,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:24,601 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:24,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:24,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:24,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,602 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,604 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,604 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,604 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,604 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:24,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:24,605 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:24,605 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,605 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:24,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:24,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:24,607 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,607 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:24,607 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:24,607 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:24,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:24,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:24,607 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:24,609 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:24,612 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:24,612 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7812963a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:24,613 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,614 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:24,614 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:24,614 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:24,614 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,614 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:24,615 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:24,615 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,615 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,615 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:24,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:24,618 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:24,618 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:24,618 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:24,618 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:24,618 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:24,621 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,621 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,621 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,629 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:24,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:24,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,630 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:24,630 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:24,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,630 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:24,730 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:24,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:24,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:24,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:24,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:24,733 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:24,733 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,733 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:24,733 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:24,734 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,734 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,735 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,735 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,735 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,735 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:24,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:24,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:24,737 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,737 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:24,737 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:24,737 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:24,737 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:24,739 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,739 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76eedc2c[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:24,741 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:24,742 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,742 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,743 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:24,743 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:24,743 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,743 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:24,743 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:24,744 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,744 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,744 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,744 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:24,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:24,753 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:24,753 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:24,753 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:24,753 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:24,753 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:24,754 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,755 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,755 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,756 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,758 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,761 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,761 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,761 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,761 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,761 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,761 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,761 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:24,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:24,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,761 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,762 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:24,762 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:24,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:24,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490524762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490524763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490524763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490524763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:24,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490524764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,819 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403d753a7bbdf9e42e69882b6e19ec0b73e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d753a7bbdf9e42e69882b6e19ec0b73e 2023-04-03 02:54:24,825 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=189, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/7e1bbfc7bc424c9387d2e1ab69c63a11 2023-04-03 02:54:24,862 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:24,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:24,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:24,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:24,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:24,865 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:24,865 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,865 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:24,865 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:24,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,867 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,867 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,868 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,868 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:24,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:24,869 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:24,869 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,869 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:24,869 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:24,869 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:24,869 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,869 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:24,869 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:24,870 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:24,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:24,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:24,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:24,871 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,871 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:24,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5071af1b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:24,873 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:24,873 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,877 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,877 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:24,878 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:24,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:24,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:24,879 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:24,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:24,882 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:24,882 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:24,882 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:24,882 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:24,882 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:24,883 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,884 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,884 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,885 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,885 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,887 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,887 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,887 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,887 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:24,887 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:24,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:24,887 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,888 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,888 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,888 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:24,888 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:24,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:24,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:24,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:24,988 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:24,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:24,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:24,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:24,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:24,992 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:24,992 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,992 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:24,992 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:24,992 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,992 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,993 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:24,993 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:24,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:24,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:24,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:24,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:24,995 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,995 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:24,995 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:24,995 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:24,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:24,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:24,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:24,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:24,997 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:24,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:24,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:24,999 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ad1afd[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:24,999 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,000 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,000 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,000 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:25,002 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:25,002 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,002 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,002 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:25,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:25,005 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:25,005 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:25,005 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:25,005 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:25,005 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:25,006 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,006 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,006 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,008 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,013 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:25,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:25,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:25,014 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:25,014 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:25,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:25,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490525069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:25,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490525070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490525070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490525071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:25,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490525075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,114 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:25,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:25,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:25,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:25,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:25,117 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:25,117 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,117 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:25,117 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:25,117 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,118 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,119 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,120 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,120 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:25,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:25,121 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,121 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:25,121 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:25,121 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:25,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,123 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,123 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,123 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:25,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@962f334[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:25,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:25,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,125 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:25,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:25,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:25,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:25,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:25,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:25,129 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:25,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:25,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:25,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:25,129 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:25,131 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,131 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,131 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,132 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,133 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,137 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:25,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:25,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:25,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,138 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:25,138 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:25,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,237 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/d41f6cf4923848008f9c856e8bc5bf7c 2023-04-03 02:54:25,238 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:25,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:25,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:25,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:25,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:25,241 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:25,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:25,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:25,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,243 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,243 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,243 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:25,245 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:25,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:25,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:25,248 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,248 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:25,248 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:25,248 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:25,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:25,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:25,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:25,252 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:25,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@96d4188[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:25,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:25,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,256 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:25,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:25,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:25,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,258 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:25,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:25,260 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:25,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:25,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:25,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:25,260 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:25,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,262 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:25,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,267 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:25,267 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:25,267 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:25,367 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:25,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:25,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:25,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:25,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:25,371 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:25,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:25,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:25,372 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,372 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,373 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,374 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:25,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:25,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:25,378 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,378 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:25,378 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:25,378 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:25,379 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:25,379 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,379 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:25,379 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:25,381 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,381 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,382 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,382 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:25,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:25,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54b93ec9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:25,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,385 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:25,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:25,386 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,386 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:25,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:25,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:25,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:25,390 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:25,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:25,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:25,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:25,390 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:25,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,394 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,400 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,400 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,400 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:25,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:25,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,401 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,401 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:25,401 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:25,401 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,401 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,401 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,401 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,401 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,401 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:25,501 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:25,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:25,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:25,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:25,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:25,514 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:25,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:25,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:25,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,517 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,517 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,517 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,517 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:25,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:25,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:25,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:25,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:25,520 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:25,520 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,520 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:25,524 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,524 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:25,524 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:25,524 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:25,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:25,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:25,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59ceac51[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:25,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:25,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:25,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:25,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:25,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,533 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:25,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:25,535 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:25,535 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:25,535 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:25,535 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:25,535 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:25,538 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,538 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,538 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,539 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,546 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,549 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:25,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:25,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,549 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:25,549 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:25,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:25,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490525571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:25,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490525574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:25,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490525579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:25,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490525579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:25,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490525579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,650 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:25,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:25,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:25,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:25,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:25,653 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:25,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:25,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:25,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,654 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/28723b37240046629e7a87e6a7339735 2023-04-03 02:54:25,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,654 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:25,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:25,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:25,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:25,657 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,657 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:25,657 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:25,657 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:25,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:25,659 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,659 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,659 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,660 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/7e1bbfc7bc424c9387d2e1ab69c63a11 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/7e1bbfc7bc424c9387d2e1ab69c63a11 2023-04-03 02:54:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:25,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@472b1f0f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:25,662 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:25,662 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,664 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:25,664 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:25,665 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,665 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:25,666 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/7e1bbfc7bc424c9387d2e1ab69c63a11, entries=350, sequenceid=189, filesize=52.6 K 2023-04-03 02:54:25,666 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:25,666 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,666 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:25,668 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:25,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:25,671 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:25,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:25,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:25,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:25,671 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:25,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/d41f6cf4923848008f9c856e8bc5bf7c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d41f6cf4923848008f9c856e8bc5bf7c 2023-04-03 02:54:25,673 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,673 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,673 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:25,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,674 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:25,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,676 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,677 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d41f6cf4923848008f9c856e8bc5bf7c, entries=150, sequenceid=189, filesize=11.8 K 2023-04-03 02:54:25,678 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/28723b37240046629e7a87e6a7339735 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/28723b37240046629e7a87e6a7339735 2023-04-03 02:54:25,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:25,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,679 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:25,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:25,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:25,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:25,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:25,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,680 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:25,680 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:25,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:25,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/28723b37240046629e7a87e6a7339735, entries=150, sequenceid=189, filesize=11.8 K 2023-04-03 02:54:25,684 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 03f67b48473b9761ac7a543789808247 in 1297ms, sequenceid=189, compaction requested=true 2023-04-03 02:54:25,684 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:25,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:25,684 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 02:54:25,685 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 02:54:25,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:25,685 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:25,687 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 183330 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 02:54:25,688 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72206 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 02:54:25,688 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/A is initiating minor compaction (all files) 2023-04-03 02:54:25,688 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/B is initiating minor compaction (all files) 2023-04-03 02:54:25,688 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/A in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,688 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/B in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,688 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a0d8bceb1ca04bcdb9a5e5c9f57c78fc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/0ac1d0d5e03a4aa6bf94a56ab46a72ce, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/93754b009f3b4a18b5a4cc532a116eb4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d1ccf912cca348e5b6276565b0c8925d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3be4fed221cf49fea053da487e12a37a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/7e1bbfc7bc424c9387d2e1ab69c63a11] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=179.0 K 2023-04-03 02:54:25,688 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/8ddd6c89d3734f9c857465390761c6d2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/94cb3c513bce486d95f4066fd850c636, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/f4b1cb10fcc9454aaa89cd31f48ba74b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/62d77a9ac9b841b7b8529e06db9af5c2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/749f9735289f42b9b24cbe86bcae0298, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d41f6cf4923848008f9c856e8bc5bf7c] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=70.5 K 2023-04-03 02:54:25,688 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a0d8bceb1ca04bcdb9a5e5c9f57c78fc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680490460148 2023-04-03 02:54:25,689 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 8ddd6c89d3734f9c857465390761c6d2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680490460148 2023-04-03 02:54:25,689 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 0ac1d0d5e03a4aa6bf94a56ab46a72ce, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680490460480 2023-04-03 02:54:25,689 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 94cb3c513bce486d95f4066fd850c636, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680490460480 2023-04-03 02:54:25,689 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 93754b009f3b4a18b5a4cc532a116eb4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680490461110 2023-04-03 02:54:25,690 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f4b1cb10fcc9454aaa89cd31f48ba74b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680490461110 2023-04-03 02:54:25,690 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting d1ccf912cca348e5b6276565b0c8925d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680490462210 2023-04-03 02:54:25,690 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 62d77a9ac9b841b7b8529e06db9af5c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680490462210 2023-04-03 02:54:25,690 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 3be4fed221cf49fea053da487e12a37a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1680490463244 2023-04-03 02:54:25,691 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 7e1bbfc7bc424c9387d2e1ab69c63a11, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680490464374 2023-04-03 02:54:25,691 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 749f9735289f42b9b24cbe86bcae0298, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1680490463244 2023-04-03 02:54:25,691 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d41f6cf4923848008f9c856e8bc5bf7c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680490464378 2023-04-03 02:54:25,714 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#B#compaction#154 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:25,720 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#A#compaction#153 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:25,780 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:25,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:25,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:25,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:25,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:25,784 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:25,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:25,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:25,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,786 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:25,786 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,787 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:25,787 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:25,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:25,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:25,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:25,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:25,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:25,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:25,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:25,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:25,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:25,788 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:25,788 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:25,788 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 02:54:25,789 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:25,789 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:25,789 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:25,789 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:25,789 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:25,789 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:25,805 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403436d7fd6a0b341e4a3f4135ef77d62f0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403436d7fd6a0b341e4a3f4135ef77d62f0 2023-04-03 02:54:26,132 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/9b4995537e7e4cf7b40708153bd95de1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/9b4995537e7e4cf7b40708153bd95de1 2023-04-03 02:54:26,136 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 03f67b48473b9761ac7a543789808247/B of 03f67b48473b9761ac7a543789808247 into 9b4995537e7e4cf7b40708153bd95de1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:26,137 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:26,137 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/B, priority=10, startTime=1680490465684; duration=0sec 2023-04-03 02:54:26,137 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:26,137 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 02:54:26,139 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72206 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 02:54:26,139 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/C is initiating minor compaction (all files) 2023-04-03 02:54:26,139 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/C in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,139 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/27f16b30f1014082999238a33d3d2c4e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/0fc0716601424e70b88e27d06addf2b9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/add2905a180248d28995981e6f34dad7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/f810cc6ef021415dbe6df1f00dd47628, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/bfcc7e1c22c14b549262d3d2eadf3caa, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/28723b37240046629e7a87e6a7339735] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=70.5 K 2023-04-03 02:54:26,140 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 27f16b30f1014082999238a33d3d2c4e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680490460148 2023-04-03 02:54:26,140 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 0fc0716601424e70b88e27d06addf2b9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680490460480 2023-04-03 02:54:26,141 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting add2905a180248d28995981e6f34dad7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680490461110 2023-04-03 02:54:26,142 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f810cc6ef021415dbe6df1f00dd47628, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680490462210 2023-04-03 02:54:26,143 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting bfcc7e1c22c14b549262d3d2eadf3caa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1680490463244 2023-04-03 02:54:26,143 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 28723b37240046629e7a87e6a7339735, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680490464378 2023-04-03 02:54:26,157 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#C#compaction#156 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:26,164 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/b54aa1a9f8e24c3281bde438e0046237 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/b54aa1a9f8e24c3281bde438e0046237 2023-04-03 02:54:26,172 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 03f67b48473b9761ac7a543789808247/A of 03f67b48473b9761ac7a543789808247 into b54aa1a9f8e24c3281bde438e0046237(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:26,172 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:26,172 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/A, priority=10, startTime=1680490465684; duration=0sec 2023-04-03 02:54:26,173 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:26,183 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/9a9c9bb218c14f0e93cb57455bb493b2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/9a9c9bb218c14f0e93cb57455bb493b2 2023-04-03 02:54:26,189 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 03f67b48473b9761ac7a543789808247/C of 03f67b48473b9761ac7a543789808247 into 9a9c9bb218c14f0e93cb57455bb493b2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:26,189 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:26,189 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/C, priority=10, startTime=1680490465685; duration=0sec 2023-04-03 02:54:26,190 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:26,214 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=226, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/99311577bd0e4055ba26994faf5e9aa2 2023-04-03 02:54:26,225 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/3ff328e0aba945c18b858595a83186dc 2023-04-03 02:54:26,237 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/022b82aa567640ee9b287329fe665d99 2023-04-03 02:54:26,241 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/99311577bd0e4055ba26994faf5e9aa2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/99311577bd0e4055ba26994faf5e9aa2 2023-04-03 02:54:26,252 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/99311577bd0e4055ba26994faf5e9aa2, entries=150, sequenceid=226, filesize=25.3 K 2023-04-03 02:54:26,253 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/3ff328e0aba945c18b858595a83186dc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/3ff328e0aba945c18b858595a83186dc 2023-04-03 02:54:26,256 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/3ff328e0aba945c18b858595a83186dc, entries=150, sequenceid=226, filesize=11.8 K 2023-04-03 02:54:26,257 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/022b82aa567640ee9b287329fe665d99 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/022b82aa567640ee9b287329fe665d99 2023-04-03 02:54:26,266 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/022b82aa567640ee9b287329fe665d99, entries=150, sequenceid=226, filesize=11.8 K 2023-04-03 02:54:26,267 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 03f67b48473b9761ac7a543789808247 in 479ms, sequenceid=226, compaction requested=false 2023-04-03 02:54:26,267 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:26,267 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:26,269 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24296763[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:26,272 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:26,272 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,273 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:26,273 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,273 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:26,274 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,274 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:26,275 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,275 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:26,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:26,278 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:26,278 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:26,278 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:26,278 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:26,278 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:26,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,281 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,285 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:26,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:26,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:26,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:26,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,286 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:26,286 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:26,286 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,286 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,286 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,286 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:26,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,386 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:26,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:26,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:26,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:26,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:26,390 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:26,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:26,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:26,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:26,391 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,391 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:26,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:26,393 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,393 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,393 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:26,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:26,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:26,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:26,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:26,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:26,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:26,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:26,395 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,395 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:26,395 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:26,395 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,395 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:26,395 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:26,395 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,395 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:26,395 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:26,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,397 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,397 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,398 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:26,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@397b4a59[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:26,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:26,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,405 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,406 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:26,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:26,406 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:26,406 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:26,407 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:26,407 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,407 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,407 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:26,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:26,410 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:26,410 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:26,410 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:26,411 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:26,411 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:26,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,413 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,417 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,417 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,417 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,417 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:26,417 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:26,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:26,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:26,417 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,417 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,417 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,417 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:26,417 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:26,418 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:26,417 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:26,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,518 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:26,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:26,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:26,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:26,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:26,521 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:26,521 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:26,521 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:26,521 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:26,522 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,522 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,524 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:26,524 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:26,525 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,525 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:26,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:26,526 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:26,526 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:26,526 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:26,526 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:26,527 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:26,527 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,527 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:26,527 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:26,527 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,527 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:26,527 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:26,527 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,527 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:26,527 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:26,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,529 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,529 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@157bd503[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:26,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:26,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,532 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:26,533 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,533 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:26,534 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,535 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:26,535 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,535 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:26,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:26,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:26,538 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:26,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:26,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:26,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:26,538 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:26,541 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,541 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,541 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,542 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,546 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:26,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,547 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:26,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:26,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:26,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:26,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,547 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:26,547 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:26,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:26,619 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:54:26,620 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:26,620 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:26,620 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:26,620 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:26,620 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:26,620 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:26,647 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:26,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:26,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:26,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:26,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:26,652 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:26,652 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:26,652 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:26,652 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:26,652 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,652 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,654 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:26,654 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:26,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:26,654 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490526649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680490466620, max timestamp=1680490466654Num uniques 859; 2023-04-03 02:54:26,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-03 02:54:26,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680490466620, max timestamp=1680490466654Num uniques 859; 2023-04-03 02:54:26,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490526653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-03 02:54:26,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680490466620, max timestamp=1680490466654Num uniques 859; 2023-04-03 02:54:26,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-03 02:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:26,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:26,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:26,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:26,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:26,658 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:26,658 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:26,658 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,658 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:26,658 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:26,658 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,658 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:26,658 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:26,658 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,658 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:26,658 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:26,659 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,660 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,660 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,660 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:26,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a3763c4[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:26,662 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:26,662 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,664 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,664 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:26,664 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:26,664 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:26,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,665 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:26,665 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,665 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:26,666 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,666 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490526664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:26,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:26,668 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:26,668 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:26,668 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:26,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:26,669 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:26,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490526664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:26,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:26,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:26,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:26,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:26,675 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,676 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:26,676 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:26,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490526675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490526757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490526757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490526768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490526770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,776 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:26,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:26,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:26,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:26,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:26,779 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:26,779 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:26,779 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:26,779 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:26,779 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,780 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,782 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:26,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:26,782 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:26,782 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,782 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:26,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:26,783 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:26,783 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:26,783 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:26,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:26,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490526780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,784 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,784 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:26,784 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:26,784 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:26,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:26,786 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,786 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,786 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,786 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ca1f9e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:26,788 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:26,788 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,789 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:26,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:26,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:26,789 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,790 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:26,791 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,791 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,791 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,791 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:26,793 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:26,793 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:26,793 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:26,793 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:26,793 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:26,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,799 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,801 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,804 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:26,804 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:26,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,806 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,806 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,806 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,806 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:26,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:26,806 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,806 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:26,806 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,807 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,807 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:26,807 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:26,907 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:26,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:26,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:26,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:26,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:26,911 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:26,911 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:26,911 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:26,911 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:26,911 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,911 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,912 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,912 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:26,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:26,913 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,913 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:26,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:26,914 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:26,914 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:26,914 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:26,914 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:26,914 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:26,914 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,914 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:26,915 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:26,915 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:26,915 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:26,915 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:26,915 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,915 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:26,915 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:26,916 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:26,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680b8fc[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:26,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:26,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,921 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,921 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:26,921 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:26,921 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:26,922 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,922 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:26,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:26,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,923 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:26,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:26,928 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:26,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:26,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:26,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:26,928 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:26,930 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,930 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:26,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,930 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:26,931 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,932 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:26,935 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:26,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:26,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:26,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:26,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:26,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:26,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:26,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:26,935 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:26,935 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:26,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:26,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490526959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490526960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490526971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490526973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:26,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:26,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490526985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,035 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:27,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:27,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:27,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:27,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:27,040 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:27,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:27,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:27,041 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,041 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,043 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:27,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:27,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:27,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:27,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:27,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:27,046 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,046 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:27,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:27,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d98a6e2[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:27,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,049 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:27,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:27,051 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:27,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:27,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:27,053 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:27,054 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:27,054 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:27,054 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:27,054 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:27,055 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040341e2d6860f034c3483440dd3e1c0e8ad to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040341e2d6860f034c3483440dd3e1c0e8ad 2023-04-03 02:54:27,057 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,057 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,057 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,058 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,060 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:27,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:27,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:27,064 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:27,064 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:27,164 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:27,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:27,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:27,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:27,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:27,172 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:27,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:27,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:27,173 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,173 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,174 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:27,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:27,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:27,176 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,176 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:27,176 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:27,176 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:27,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:27,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c80d5d6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:27,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:27,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:27,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:27,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:27,186 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,186 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:27,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:27,188 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:27,189 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:27,189 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:27,189 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:27,189 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:27,190 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,190 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,190 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,192 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,199 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:27,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:27,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:27,204 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:27,204 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:27,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:27,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490527262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:27,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490527263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:27,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490527275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:27,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490527279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:27,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490527288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,304 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:27,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:27,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:27,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:27,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:27,308 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:27,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:27,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:27,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,309 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,310 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,310 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:27,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:27,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:27,311 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,311 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:27,311 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:27,311 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:27,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:27,313 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:27,314 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:27,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49831337[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:27,314 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,315 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,315 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:27,315 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:27,315 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:27,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:27,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:27,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:27,325 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:27,325 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:27,325 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:27,325 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:27,325 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:27,327 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,327 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,327 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,328 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,329 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,332 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:27,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:27,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:27,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,332 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:27,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,332 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:27,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,433 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:27,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:27,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:27,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:27,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:27,435 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:27,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:27,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:27,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,438 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,438 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,438 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,438 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:27,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:27,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:27,439 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,439 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:27,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:27,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:27,442 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,443 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,443 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:27,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ac715a3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:27,444 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:27,444 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,445 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:27,445 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,445 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:27,447 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:27,447 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,447 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:27,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:27,449 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:27,449 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:27,449 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:27,449 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:27,449 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:27,450 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,451 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,451 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,451 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,452 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:27,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:27,455 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,456 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,456 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:27,456 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,456 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:27,456 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,456 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:27,456 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,462 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=242, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/34c2175f20784e98afe5e422545945b4 2023-04-03 02:54:27,556 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:27,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:27,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:27,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:27,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:27,559 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:27,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:27,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:27,560 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,560 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:27,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:27,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:27,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:27,564 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,564 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:27,564 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:27,564 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:27,566 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:27,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@426015af[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:27,569 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:27,569 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,570 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:27,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:27,576 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,576 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:27,576 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,576 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:27,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:27,578 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:27,578 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:27,578 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:27,578 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:27,578 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:27,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,580 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,582 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,586 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,586 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,586 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,586 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,586 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,586 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,586 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,586 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,586 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:27,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:27,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,586 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:27,587 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:27,587 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:27,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,687 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:27,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:27,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:27,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:27,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:27,692 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:27,692 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,692 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:27,692 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:27,697 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,698 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,700 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,701 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,701 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:27,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:27,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:27,702 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:27,702 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,702 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:27,702 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:27,703 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:27,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:27,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:27,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:27,704 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,704 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,705 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,705 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:27,706 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:27,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38b44650[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:27,706 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,707 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,707 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:27,707 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:27,707 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:27,707 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,707 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:27,709 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:27,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:27,711 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:27,711 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:27,711 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:27,711 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:27,711 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:27,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,716 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,717 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,719 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,719 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,719 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:27,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:27,719 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,719 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,719 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:27,720 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:27,720 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,720 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,720 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,720 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,720 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:27,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:27,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490527765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:27,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490527771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:27,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490527781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:27,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490527785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:27,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490527796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,820 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:27,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:27,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:27,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:27,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:27,823 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:27,823 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,823 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:27,823 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:27,824 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,824 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,825 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,825 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:27,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:27,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:27,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:27,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:27,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:27,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:27,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:27,827 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,827 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:27,827 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:27,827 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:27,828 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:27,829 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@697ff0da[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:27,831 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:27,831 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,835 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,835 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:27,835 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:27,835 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:27,835 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,835 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:27,836 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:27,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:27,839 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:27,839 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:27,839 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:27,839 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:27,839 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:27,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:27,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:27,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,841 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:27,843 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,845 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,845 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,845 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:27,845 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,845 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,845 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,846 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:27,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:27,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:27,846 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,846 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:27,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,846 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:27,846 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,846 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:27,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:27,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:27,873 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/474a36a6b07e4084a9a231b969cce83c 2023-04-03 02:54:27,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/127643bc2aa6435d8a25ba08e872daac 2023-04-03 02:54:27,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/34c2175f20784e98afe5e422545945b4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/34c2175f20784e98afe5e422545945b4 2023-04-03 02:54:27,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/34c2175f20784e98afe5e422545945b4, entries=400, sequenceid=242, filesize=59.4 K 2023-04-03 02:54:27,893 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/474a36a6b07e4084a9a231b969cce83c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/474a36a6b07e4084a9a231b969cce83c 2023-04-03 02:54:27,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/474a36a6b07e4084a9a231b969cce83c, entries=150, sequenceid=242, filesize=11.8 K 2023-04-03 02:54:27,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/127643bc2aa6435d8a25ba08e872daac as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/127643bc2aa6435d8a25ba08e872daac 2023-04-03 02:54:27,902 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/127643bc2aa6435d8a25ba08e872daac, entries=150, sequenceid=242, filesize=11.8 K 2023-04-03 02:54:27,903 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=154.31 KB/158010 for 03f67b48473b9761ac7a543789808247 in 1284ms, sequenceid=242, compaction requested=true 2023-04-03 02:54:27,903 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:27,903 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:27,903 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:27,904 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:27,903 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:27,904 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:27,905 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:27,905 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/B is initiating minor compaction (all files) 2023-04-03 02:54:27,905 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113090 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:27,905 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/B in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,905 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/A is initiating minor compaction (all files) 2023-04-03 02:54:27,905 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/9b4995537e7e4cf7b40708153bd95de1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/3ff328e0aba945c18b858595a83186dc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/474a36a6b07e4084a9a231b969cce83c] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=35.7 K 2023-04-03 02:54:27,905 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/A in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,905 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/b54aa1a9f8e24c3281bde438e0046237, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/99311577bd0e4055ba26994faf5e9aa2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/34c2175f20784e98afe5e422545945b4] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=110.4 K 2023-04-03 02:54:27,906 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 9b4995537e7e4cf7b40708153bd95de1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680490464378 2023-04-03 02:54:27,906 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting b54aa1a9f8e24c3281bde438e0046237, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680490464378 2023-04-03 02:54:27,907 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 3ff328e0aba945c18b858595a83186dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680490464443 2023-04-03 02:54:27,907 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 99311577bd0e4055ba26994faf5e9aa2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680490464443 2023-04-03 02:54:27,907 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 474a36a6b07e4084a9a231b969cce83c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680490466606 2023-04-03 02:54:27,907 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 34c2175f20784e98afe5e422545945b4, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680490466589 2023-04-03 02:54:27,924 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#B#compaction#162 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:27,928 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#A#compaction#163 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:27,946 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:27,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:27,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:27,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:27,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:27,952 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:27,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:27,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:27,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,954 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:27,954 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:27,955 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:27,955 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:27,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:27,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:27,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:27,956 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:27,956 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:27,956 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:27,956 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:27,956 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:27,956 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:27,956 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:27,956 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=154.31 KB heapSize=379.13 KB 2023-04-03 02:54:27,957 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:27,957 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:27,957 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:27,957 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:27,957 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:27,957 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:28,356 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/781ad2a1e6ff418f827f10dd6157955e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/781ad2a1e6ff418f827f10dd6157955e 2023-04-03 02:54:28,362 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 03f67b48473b9761ac7a543789808247/B of 03f67b48473b9761ac7a543789808247 into 781ad2a1e6ff418f827f10dd6157955e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:28,362 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:28,362 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/B, priority=13, startTime=1680490467903; duration=0sec 2023-04-03 02:54:28,362 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:28,362 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:28,363 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:28,364 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/C is initiating minor compaction (all files) 2023-04-03 02:54:28,364 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/C in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:28,364 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/9a9c9bb218c14f0e93cb57455bb493b2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/022b82aa567640ee9b287329fe665d99, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/127643bc2aa6435d8a25ba08e872daac] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=35.7 K 2023-04-03 02:54:28,364 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 9a9c9bb218c14f0e93cb57455bb493b2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680490464378 2023-04-03 02:54:28,365 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 022b82aa567640ee9b287329fe665d99, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680490464443 2023-04-03 02:54:28,365 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 127643bc2aa6435d8a25ba08e872daac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680490466606 2023-04-03 02:54:28,373 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403d74b5245784942f69c17c0b7d07451a7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d74b5245784942f69c17c0b7d07451a7 2023-04-03 02:54:28,377 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#C#compaction#165 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:28,760 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/9ca002d54b4947c68d4ea238c192b652 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/9ca002d54b4947c68d4ea238c192b652 2023-04-03 02:54:28,765 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 03f67b48473b9761ac7a543789808247/A of 03f67b48473b9761ac7a543789808247 into 9ca002d54b4947c68d4ea238c192b652(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:28,765 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:28,765 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/A, priority=13, startTime=1680490467903; duration=0sec 2023-04-03 02:54:28,765 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:28,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:28,778 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:28,784 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=268, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/6d575df901004114ae114517781a306a 2023-04-03 02:54:28,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:28,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490528790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:28,791 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/b8e8a85b10b44f21b01ffb014054ba08 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/b8e8a85b10b44f21b01ffb014054ba08 2023-04-03 02:54:28,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:28,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490528791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:28,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:28,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490528791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:28,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:28,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490528793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:28,796 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 03f67b48473b9761ac7a543789808247/C of 03f67b48473b9761ac7a543789808247 into b8e8a85b10b44f21b01ffb014054ba08(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:28,796 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:28,796 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/C, priority=13, startTime=1680490467904; duration=0sec 2023-04-03 02:54:28,796 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:28,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:28,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490528808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:28,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:28,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490528892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:28,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:28,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490528894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:28,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:28,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490528898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:29,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490529095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:29,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490529097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:29,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490529103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,199 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/d97304f9f9514691a27b65981d3a1ac7 2023-04-03 02:54:29,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:29,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60330 deadline: 1680490529398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:29,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490529401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:29,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490529405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,619 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/e2d0c5fbe074404aa9b72e80819c4d83 2023-04-03 02:54:29,624 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/6d575df901004114ae114517781a306a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6d575df901004114ae114517781a306a 2023-04-03 02:54:29,630 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6d575df901004114ae114517781a306a, entries=150, sequenceid=268, filesize=25.4 K 2023-04-03 02:54:29,631 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/d97304f9f9514691a27b65981d3a1ac7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d97304f9f9514691a27b65981d3a1ac7 2023-04-03 02:54:29,635 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d97304f9f9514691a27b65981d3a1ac7, entries=150, sequenceid=268, filesize=11.9 K 2023-04-03 02:54:29,637 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/e2d0c5fbe074404aa9b72e80819c4d83 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/e2d0c5fbe074404aa9b72e80819c4d83 2023-04-03 02:54:29,645 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/e2d0c5fbe074404aa9b72e80819c4d83, entries=150, sequenceid=268, filesize=11.9 K 2023-04-03 02:54:29,646 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~379.08 KB/388176, currentSize=53.67 KB/54960 for 03f67b48473b9761ac7a543789808247 in 1690ms, sequenceid=268, compaction requested=false 2023-04-03 02:54:29,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:29,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:29,649 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,649 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:29,650 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:29,650 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:29,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:29,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1be78a5b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:29,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:29,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:29,655 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:29,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:29,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:29,656 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,656 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:29,656 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:29,656 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:29,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:29,667 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:29,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:29,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:29,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:29,667 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:29,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:29,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:29,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:29,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:29,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:29,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:29,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:29,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:29,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:29,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:29,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:29,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:29,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:29,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:29,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:29,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:29,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:29,674 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:29,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:29,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:29,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:29,675 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:29,675 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:29,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:29,776 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:29,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:29,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:29,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:29,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:29,783 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:29,783 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:29,783 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:29,783 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:29,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:29,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:29,786 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,786 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:29,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:29,786 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,786 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:29,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:29,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:29,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:29,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:29,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:29,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:29,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:29,792 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:29,792 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:29,792 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:29,792 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:29,792 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 02:54:29,793 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:29,794 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:29,795 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:29,795 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:29,795 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:29,795 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:29,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:29,905 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:29,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:29,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490529925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:29,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490529926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:29,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680490469915, max timestamp=1680490469927Num uniques 859; 2023-04-03 02:54:29,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 02:54:29,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680490469915, max timestamp=1680490469927Num uniques 859; 2023-04-03 02:54:29,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 02:54:29,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680490469915, max timestamp=1680490469927Num uniques 859; 2023-04-03 02:54:29,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 02:54:29,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:29,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490529998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:30,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490530031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:30,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490530031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:30,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60390 deadline: 1680490530099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:30,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490530233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:30,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490530233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,250 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403e87987cbe34c4b6195629ac23c204ab4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e87987cbe34c4b6195629ac23c204ab4 2023-04-03 02:54:30,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:30,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60390 deadline: 1680490530300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:30,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490530535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:30,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490530536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:30,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60390 deadline: 1680490530606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,655 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=282, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/d3c68a9cdfdb444199bcaf3ddd8592dd 2023-04-03 02:54:30,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:30,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490530799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,802 DEBUG [Thread-885] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:54:30,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:30,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490530821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:30,823 DEBUG [Thread-883] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:54:31,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:31,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490531038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:31,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60386 deadline: 1680490531042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,067 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/58616957961d48b9927031036d861b77 2023-04-03 02:54:31,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:31,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60390 deadline: 1680490531112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,483 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/3111399f347f474a895bfc05c4f2c1bc 2023-04-03 02:54:31,488 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/d3c68a9cdfdb444199bcaf3ddd8592dd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d3c68a9cdfdb444199bcaf3ddd8592dd 2023-04-03 02:54:31,492 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d3c68a9cdfdb444199bcaf3ddd8592dd, entries=150, sequenceid=282, filesize=25.4 K 2023-04-03 02:54:31,493 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/58616957961d48b9927031036d861b77 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/58616957961d48b9927031036d861b77 2023-04-03 02:54:31,498 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/58616957961d48b9927031036d861b77, entries=150, sequenceid=282, filesize=11.9 K 2023-04-03 02:54:31,499 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/3111399f347f474a895bfc05c4f2c1bc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/3111399f347f474a895bfc05c4f2c1bc 2023-04-03 02:54:31,503 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/3111399f347f474a895bfc05c4f2c1bc, entries=150, sequenceid=282, filesize=11.9 K 2023-04-03 02:54:31,504 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 03f67b48473b9761ac7a543789808247 in 1712ms, sequenceid=282, compaction requested=true 2023-04-03 02:54:31,504 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:31,504 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:31,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:31,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:31,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:31,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:31,510 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,510 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:31,510 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:31,510 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:31,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:31,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e829e5[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:31,512 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:31,512 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:31,513 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:31,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:31,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:31,513 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:31,513 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:31,513 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:31,513 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,513 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:31,514 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,514 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:31,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:31,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:31,514 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:31,515 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:31,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:31,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:31,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:31,518 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:31,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:31,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:31,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:31,518 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:31,526 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:31,526 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:31,527 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:31,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:31,527 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:31,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:31,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:31,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:31,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:31,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:31,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:31,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,531 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:31,531 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:31,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:31,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:31,532 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:31,532 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,532 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:31,532 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:31,532 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:31,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:31,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:31,632 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:31,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:31,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:31,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:31,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:31,637 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:31,637 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:31,637 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:31,637 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:31,637 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:31,637 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:31,639 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:31,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:31,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:31,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:31,639 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:31,640 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:31,640 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:31,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:31,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:31,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:31,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:31,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:31,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:31,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:31,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:31,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:31,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:31,641 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:31,641 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:31,641 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-03 02:54:31,642 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:31,642 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:31,643 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:31,643 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:31,643 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:31,643 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:32,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:32,042 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:32,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:32,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490532060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:32,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:32,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490532063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:32,075 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040333dd610f7ddf4ce1971e190ede7bf1c8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040333dd610f7ddf4ce1971e190ede7bf1c8 2023-04-03 02:54:32,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:32,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60390 deadline: 1680490532122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:32,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:32,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490532165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:32,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:32,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490532168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:32,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:32,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490532367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:32,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:32,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490532371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:32,482 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=316, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/3e63a4e681a54768a0b8ca5c36245a75 2023-04-03 02:54:32,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:32,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490532671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:32,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:32,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490532674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:32,910 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/ef861fb77f0f43609829d6e76f91cdfe 2023-04-03 02:54:33,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:33,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490533177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:33,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490533178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,341 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/2f08127b26fc46f1a632d65be0360c41 2023-04-03 02:54:33,347 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/3e63a4e681a54768a0b8ca5c36245a75 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3e63a4e681a54768a0b8ca5c36245a75 2023-04-03 02:54:33,351 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3e63a4e681a54768a0b8ca5c36245a75, entries=150, sequenceid=316, filesize=25.4 K 2023-04-03 02:54:33,352 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/ef861fb77f0f43609829d6e76f91cdfe as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/ef861fb77f0f43609829d6e76f91cdfe 2023-04-03 02:54:33,360 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/ef861fb77f0f43609829d6e76f91cdfe, entries=150, sequenceid=316, filesize=11.9 K 2023-04-03 02:54:33,360 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/2f08127b26fc46f1a632d65be0360c41 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/2f08127b26fc46f1a632d65be0360c41 2023-04-03 02:54:33,364 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/2f08127b26fc46f1a632d65be0360c41, entries=150, sequenceid=316, filesize=11.9 K 2023-04-03 02:54:33,365 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 03f67b48473b9761ac7a543789808247 in 1724ms, sequenceid=316, compaction requested=true 2023-04-03 02:54:33,365 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:33,365 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:33,365 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:33,365 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:33,365 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,366 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:33,366 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:33,367 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,367 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:33,367 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:33,367 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43f4724e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:33,369 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:33,369 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:33,370 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:33,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:33,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:33,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:33,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:33,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:33,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:33,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:33,374 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:33,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:33,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:33,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:33,377 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:33,377 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:33,377 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:33,377 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:33,377 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:33,378 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:33,378 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:33,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:33,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:33,379 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:33,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:33,379 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:33,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:33,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:33,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,382 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,384 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,385 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:33,385 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:33,385 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:33,385 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:33,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:33,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:33,385 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,385 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:33,385 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:33,385 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:33,385 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:33,385 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:33,385 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:33,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:33,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:33,485 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:33,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:33,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:33,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:33,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:33,489 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:33,489 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:33,489 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:33,489 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:33,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:33,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:33,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:33,492 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:33,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:33,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:33,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:33,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:33,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:33,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:33,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:33,499 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:33,499 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:33,499 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:33,500 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:33,500 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:33,500 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:33,500 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:33,500 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 02:54:33,500 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:33,500 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:33,500 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:33,500 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:33,500 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:33,501 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:33,542 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403eadbbb5286524cadbdc36632fef3bdeb to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403eadbbb5286524cadbdc36632fef3bdeb 2023-04-03 02:54:33,554 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=327, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/c26ab44ae50e4abd9b7139d916b7eb3f 2023-04-03 02:54:33,974 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/83caedcd1e154707ad6c5cb237d7c2da 2023-04-03 02:54:34,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:34,151 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:34,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490474183, max timestamp=1680490474192Num uniques 859; 2023-04-03 02:54:34,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:54:34,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490474183, max timestamp=1680490474192Num uniques 859; 2023-04-03 02:54:34,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:54:34,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490474183, max timestamp=1680490474192Num uniques 859; 2023-04-03 02:54:34,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:54:34,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60390 deadline: 1680490534217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490534217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490534219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60390 deadline: 1680490534323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490534323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490534324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,386 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/d8851715710747fea58807415c11537f 2023-04-03 02:54:34,390 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/c26ab44ae50e4abd9b7139d916b7eb3f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/c26ab44ae50e4abd9b7139d916b7eb3f 2023-04-03 02:54:34,397 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/c26ab44ae50e4abd9b7139d916b7eb3f, entries=150, sequenceid=327, filesize=25.4 K 2023-04-03 02:54:34,398 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/83caedcd1e154707ad6c5cb237d7c2da as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/83caedcd1e154707ad6c5cb237d7c2da 2023-04-03 02:54:34,402 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/83caedcd1e154707ad6c5cb237d7c2da, entries=150, sequenceid=327, filesize=11.9 K 2023-04-03 02:54:34,402 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/d8851715710747fea58807415c11537f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d8851715710747fea58807415c11537f 2023-04-03 02:54:34,406 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d8851715710747fea58807415c11537f, entries=150, sequenceid=327, filesize=11.9 K 2023-04-03 02:54:34,407 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 03f67b48473b9761ac7a543789808247 in 907ms, sequenceid=327, compaction requested=true 2023-04-03 02:54:34,407 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:34,407 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:34,407 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:34,407 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:34,407 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,407 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:34,407 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:34,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,410 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,410 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:34,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:34,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@551ba3ca[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:34,412 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:34,412 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,413 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:34,413 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:34,413 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:34,413 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,413 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:34,419 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:34,419 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,419 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:34,419 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:34,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:34,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:34,421 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:34,421 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:34,421 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:34,422 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:34,422 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:34,423 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,423 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,423 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,424 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,425 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:34,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:34,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:34,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:34,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:34,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:34,428 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:34,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:34,429 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,429 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:34,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:34,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:34,429 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,429 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:34,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:34,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-03 02:54:34,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:34,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:34,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:34,529 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:34,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:34,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:34,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:34,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:34,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:34,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:34,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:34,533 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:34,533 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:34,533 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:34,533 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:34,533 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,534 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,535 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,535 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:34,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:34,535 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,535 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:34,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:34,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:34,536 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:34,536 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:34,537 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:34,537 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:34,537 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:34,537 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:34,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:34,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490534538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72894e60[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:34,543 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:34,543 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490534541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60390 deadline: 1680490534543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,548 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:34,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:34,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:34,548 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,548 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:34,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,549 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:34,549 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,549 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:34,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:34,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:34,552 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:34,552 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:34,552 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:34,552 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:34,552 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:34,553 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,554 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,555 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:34,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:34,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:34,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,558 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:34,558 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:34,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:34,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:34,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:34,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:34,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:34,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:34,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:34,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490534647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490534647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60390 deadline: 1680490534648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,658 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:34,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:34,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:34,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:34,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:34,664 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:34,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:34,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:34,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:34,665 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,665 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,666 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:34,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:34,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:34,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:34,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:34,669 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:34,669 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:34,669 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:34,669 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:34,669 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:34,670 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:34,670 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:34,670 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:34,670 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:34,670 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:34,670 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:34,670 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,670 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:34,670 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:34,672 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:34,674 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:34,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ab6e6d5[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:34,674 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,677 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:34,677 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,677 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:34,678 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:34,678 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:34,678 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,678 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:34,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:34,681 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:34,681 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:34,682 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:34,682 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:34,682 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:34,684 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,684 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,684 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:34,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,691 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,691 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,691 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,691 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,691 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:34,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:34,691 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:34,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:34,691 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:34,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:34,691 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,691 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:34,691 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:34,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:34,791 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:34,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:34,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:34,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:34,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:34,795 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:34,795 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:34,795 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:34,795 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:34,795 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,796 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:34,798 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:34,798 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,798 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:34,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:34,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:34,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:34,799 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:34,799 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:34,799 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:34,799 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:34,799 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:34,799 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:34,800 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:34,800 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:34,800 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:34,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:34,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:34,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:34,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:34,802 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:34,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:34,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:34,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a309fcf[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:34,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,805 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:34,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:34,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:34,805 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,805 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:34,806 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,806 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:34,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,806 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:34,807 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:34,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:34,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:34,809 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:34,809 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:34,809 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:34,809 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:34,809 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:34,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:34,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,811 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:34,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:34,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,813 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:34,815 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:34,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:34,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:34,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:34,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:34,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,816 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:34,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:34,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:34,816 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:34,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60346 deadline: 1680490534841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60402 deadline: 1680490534842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,847 DEBUG [Thread-883] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:54:34,848 DEBUG [Thread-885] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:54:34,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490534853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490534854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60390 deadline: 1680490534855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,916 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:34,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:34,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:34,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:34,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:34,919 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:34,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:34,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:34,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:34,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,922 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:34,922 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:34,922 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,922 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:34,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:34,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:34,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:34,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:34,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:34,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:34,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:34,924 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:34,924 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:34,924 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:34,924 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:34,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:34,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:34,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:34,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:34,926 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,926 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,926 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,926 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:34,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:34,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd68b0[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:34,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:34,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,930 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:34,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:34,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:34,930 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,930 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:34,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,934 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:34,934 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,934 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:34,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:34,937 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:34,937 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:34,937 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:34,937 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:34,937 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:34,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:34,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:34,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:34,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:34,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:34,939 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:34,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,940 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:34,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:34,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:34,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:34,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:34,943 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:34,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:34,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:34,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:34,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:34,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:34,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:34,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:34,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:34,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:34,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:34,943 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:34,943 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:34,950 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403334cf9054a734a459e3c12dfbd19a224 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403334cf9054a734a459e3c12dfbd19a224 2023-04-03 02:54:35,044 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:35,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:35,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:35,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:35,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:35,047 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:35,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:35,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:35,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:35,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,050 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:35,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:35,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:35,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:35,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:35,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:35,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:35,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:35,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:35,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:35,051 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:35,051 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:35,052 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:35,052 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:35,053 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,053 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:35,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:35,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:35,055 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:35,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a7fccc9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:35,055 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,056 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:35,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:35,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:35,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:35,058 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:35,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:35,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:35,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:35,061 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:35,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:35,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:35,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:35,061 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:35,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:35,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:35,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:35,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,064 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:35,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,065 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:35,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,067 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,067 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:35,067 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,067 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:35,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:35,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:35,067 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,067 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,067 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,067 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,067 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:35,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:35,067 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:35,067 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:35,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:35,067 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:35,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:35,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:35,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:35,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490535162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:35,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60390 deadline: 1680490535162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:35,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490535163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,168 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:35,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:35,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:35,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:35,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:35,174 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:35,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:35,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:35,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:35,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:35,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:35,177 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,177 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,177 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:35,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:35,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:35,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:35,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:35,178 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:35,178 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:35,179 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:35,179 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:35,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:35,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:35,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:35,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:35,181 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:35,181 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:35,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:35,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:35,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:35,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67a2151d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:35,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,185 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:35,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:35,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:35,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:35,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:35,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:35,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:35,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:35,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:35,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:35,190 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:35,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:35,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:35,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:35,190 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:35,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:35,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:35,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,192 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:35,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:35,194 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:35,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:35,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:35,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:35,197 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:35,197 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:35,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:35,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:35,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:35,197 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:35,197 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:35,197 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:35,297 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:35,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:35,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:35,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:35,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:35,300 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:35,300 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:35,300 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:35,301 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:35,301 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,301 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,304 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:35,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:35,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:35,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:35,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:35,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:35,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:35,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:35,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:35,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:35,308 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:35,308 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. started... 2023-04-03 02:54:35,309 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. as already flushing 2023-04-03 02:54:35,309 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:35,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:35,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:35,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:35,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:35,311 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,312 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:35,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:35,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:35,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69e220c1[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:35,318 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:35,318 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,319 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,319 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:35,319 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:35,319 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:35,319 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,319 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:35,320 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:35,321 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:35,321 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:35,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:35,324 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:35,324 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:35,324 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:35,324 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:35,324 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:35,327 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,327 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:35,327 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:35,328 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,329 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:35,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:35,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:35,337 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:35,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:35,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:35,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:35,337 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:35,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:35,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:35,338 DEBUG [Thread-887] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:35,338 DEBUG [Thread-887] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:35,356 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=363, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/00f296ef2fd64204bf837968926a57a8 2023-04-03 02:54:35,385 DEBUG [Thread-890] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:52506 2023-04-03 02:54:35,385 DEBUG [Thread-890] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:35,388 DEBUG [Thread-888] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:52506 2023-04-03 02:54:35,388 DEBUG [Thread-888] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:35,391 DEBUG [Thread-894] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2efdb02f to 127.0.0.1:52506 2023-04-03 02:54:35,391 DEBUG [Thread-894] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:35,395 DEBUG [Thread-892] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:52506 2023-04-03 02:54:35,395 DEBUG [Thread-892] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:35,438 DEBUG [Thread-887] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:35,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:35,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:35,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60386 deadline: 1680490535667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:35,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:35,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60390 deadline: 1680490535669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60330 deadline: 1680490535669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:35,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/2a3ff09bfb724288b027e57cd4a50e44 2023-04-03 02:54:36,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/1cef0b6443a74534951a35be65d2ebe2 2023-04-03 02:54:36,183 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/00f296ef2fd64204bf837968926a57a8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/00f296ef2fd64204bf837968926a57a8 2023-04-03 02:54:36,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/00f296ef2fd64204bf837968926a57a8, entries=200, sequenceid=363, filesize=32.3 K 2023-04-03 02:54:36,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/2a3ff09bfb724288b027e57cd4a50e44 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2a3ff09bfb724288b027e57cd4a50e44 2023-04-03 02:54:36,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2a3ff09bfb724288b027e57cd4a50e44, entries=150, sequenceid=363, filesize=11.9 K 2023-04-03 02:54:36,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/1cef0b6443a74534951a35be65d2ebe2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/1cef0b6443a74534951a35be65d2ebe2 2023-04-03 02:54:36,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/1cef0b6443a74534951a35be65d2ebe2, entries=150, sequenceid=363, filesize=11.9 K 2023-04-03 02:54:36,194 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 03f67b48473b9761ac7a543789808247 in 1667ms, sequenceid=363, compaction requested=true 2023-04-03 02:54:36,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:36,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:36,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:36,195 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 02:54:36,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:36,195 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 02:54:36,196 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163712 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 02:54:36,196 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/A is initiating minor compaction (all files) 2023-04-03 02:54:36,196 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73512 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 02:54:36,196 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/A in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:36,196 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/B is initiating minor compaction (all files) 2023-04-03 02:54:36,197 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/9ca002d54b4947c68d4ea238c192b652, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6d575df901004114ae114517781a306a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d3c68a9cdfdb444199bcaf3ddd8592dd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3e63a4e681a54768a0b8ca5c36245a75, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/c26ab44ae50e4abd9b7139d916b7eb3f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/00f296ef2fd64204bf837968926a57a8] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=159.9 K 2023-04-03 02:54:36,197 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/B in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:36,197 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/781ad2a1e6ff418f827f10dd6157955e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d97304f9f9514691a27b65981d3a1ac7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/58616957961d48b9927031036d861b77, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/ef861fb77f0f43609829d6e76f91cdfe, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/83caedcd1e154707ad6c5cb237d7c2da, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2a3ff09bfb724288b027e57cd4a50e44] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=71.8 K 2023-04-03 02:54:36,197 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 9ca002d54b4947c68d4ea238c192b652, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680490466606 2023-04-03 02:54:36,197 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 781ad2a1e6ff418f827f10dd6157955e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680490466606 2023-04-03 02:54:36,197 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6d575df901004114ae114517781a306a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680490466654 2023-04-03 02:54:36,197 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d97304f9f9514691a27b65981d3a1ac7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680490466654 2023-04-03 02:54:36,198 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting d3c68a9cdfdb444199bcaf3ddd8592dd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680490468785 2023-04-03 02:54:36,198 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 58616957961d48b9927031036d861b77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680490468785 2023-04-03 02:54:36,198 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 3e63a4e681a54768a0b8ca5c36245a75, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680490469942 2023-04-03 02:54:36,198 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ef861fb77f0f43609829d6e76f91cdfe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680490469942 2023-04-03 02:54:36,198 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c26ab44ae50e4abd9b7139d916b7eb3f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680490472056 2023-04-03 02:54:36,198 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 83caedcd1e154707ad6c5cb237d7c2da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680490472056 2023-04-03 02:54:36,199 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 00f296ef2fd64204bf837968926a57a8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1680490474215 2023-04-03 02:54:36,199 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 2a3ff09bfb724288b027e57cd4a50e44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1680490474215 2023-04-03 02:54:36,209 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#B#compaction#181 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:36,213 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#A#compaction#180 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:36,620 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/ab1da32711714f498a4bcb513bb989dc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/ab1da32711714f498a4bcb513bb989dc 2023-04-03 02:54:36,625 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 03f67b48473b9761ac7a543789808247/B of 03f67b48473b9761ac7a543789808247 into ab1da32711714f498a4bcb513bb989dc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:36,625 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:36,625 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/B, priority=10, startTime=1680490476195; duration=0sec 2023-04-03 02:54:36,625 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:36,625 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 02:54:36,627 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73512 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 02:54:36,628 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 03f67b48473b9761ac7a543789808247/C is initiating minor compaction (all files) 2023-04-03 02:54:36,628 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03f67b48473b9761ac7a543789808247/C in TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:36,628 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/b8e8a85b10b44f21b01ffb014054ba08, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/e2d0c5fbe074404aa9b72e80819c4d83, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/3111399f347f474a895bfc05c4f2c1bc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/2f08127b26fc46f1a632d65be0360c41, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d8851715710747fea58807415c11537f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/1cef0b6443a74534951a35be65d2ebe2] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp, totalSize=71.8 K 2023-04-03 02:54:36,628 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b8e8a85b10b44f21b01ffb014054ba08, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680490466606 2023-04-03 02:54:36,629 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e2d0c5fbe074404aa9b72e80819c4d83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680490466654 2023-04-03 02:54:36,629 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 3111399f347f474a895bfc05c4f2c1bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680490468785 2023-04-03 02:54:36,629 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 2f08127b26fc46f1a632d65be0360c41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680490469942 2023-04-03 02:54:36,630 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d8851715710747fea58807415c11537f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680490472056 2023-04-03 02:54:36,630 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 1cef0b6443a74534951a35be65d2ebe2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1680490474215 2023-04-03 02:54:36,634 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/841868d819894ff4ac93caa4d84f3e29 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/841868d819894ff4ac93caa4d84f3e29 2023-04-03 02:54:36,638 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 03f67b48473b9761ac7a543789808247/A of 03f67b48473b9761ac7a543789808247 into 841868d819894ff4ac93caa4d84f3e29(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:36,639 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:36,639 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/A, priority=10, startTime=1680490476194; duration=0sec 2023-04-03 02:54:36,639 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:36,640 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 03f67b48473b9761ac7a543789808247#C#compaction#182 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:36,648 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/e178ae67a9bb4312a5fba09437cf8bdc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/e178ae67a9bb4312a5fba09437cf8bdc 2023-04-03 02:54:36,651 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 03f67b48473b9761ac7a543789808247/C of 03f67b48473b9761ac7a543789808247 into e178ae67a9bb4312a5fba09437cf8bdc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:36,651 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:36,652 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247., storeName=03f67b48473b9761ac7a543789808247/C, priority=10, startTime=1680490476195; duration=0sec 2023-04-03 02:54:36,652 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:36,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:36,671 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:54:36,671 DEBUG [Thread-879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:52506 2023-04-03 02:54:36,671 DEBUG [Thread-879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:36,671 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:36,672 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:36,672 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:36,672 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:36,672 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:36,672 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:36,674 DEBUG [Thread-877] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21281a37 to 127.0.0.1:52506 2023-04-03 02:54:36,674 DEBUG [Thread-877] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:36,675 DEBUG [Thread-881] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:52506 2023-04-03 02:54:36,675 DEBUG [Thread-881] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:37,084 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403fceb2823c79747abaca81f82bb504e8e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fceb2823c79747abaca81f82bb504e8e 2023-04-03 02:54:37,489 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=376, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/68d1670b0b7142f6bde7068717d6de29 2023-04-03 02:54:37,899 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/5ca27ec83ee04081a61c0c65bd991d1b 2023-04-03 02:54:38,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/ef0b93af51b742ddae6c821b76f09b46 2023-04-03 02:54:38,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/68d1670b0b7142f6bde7068717d6de29 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/68d1670b0b7142f6bde7068717d6de29 2023-04-03 02:54:38,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/68d1670b0b7142f6bde7068717d6de29, entries=100, sequenceid=376, filesize=18.6 K 2023-04-03 02:54:38,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/5ca27ec83ee04081a61c0c65bd991d1b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/5ca27ec83ee04081a61c0c65bd991d1b 2023-04-03 02:54:38,319 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/5ca27ec83ee04081a61c0c65bd991d1b, entries=100, sequenceid=376, filesize=9.5 K 2023-04-03 02:54:38,320 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/ef0b93af51b742ddae6c821b76f09b46 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/ef0b93af51b742ddae6c821b76f09b46 2023-04-03 02:54:38,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/ef0b93af51b742ddae6c821b76f09b46, entries=100, sequenceid=376, filesize=9.5 K 2023-04-03 02:54:38,324 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=13.42 KB/13740 for 03f67b48473b9761ac7a543789808247 in 1653ms, sequenceid=376, compaction requested=false 2023-04-03 02:54:38,325 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:42,595 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-03 02:54:44,916 DEBUG [Thread-883] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:52506 2023-04-03 02:54:44,917 DEBUG [Thread-883] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:44,946 DEBUG [Thread-885] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:52506 2023-04-03 02:54:44,946 DEBUG [Thread-885] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:44,946 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 65 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 71 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 46 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 4861 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 5158 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 2072 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 6213 rows 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 2064 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 6192 rows 2023-04-03 02:54:44,947 INFO [Listener at localhost/46853] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 02:54:44,947 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7829b43e to 127.0.0.1:52506 2023-04-03 02:54:44,947 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:54:44,949 INFO [Listener at localhost/46853] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 02:54:44,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 02:54:44,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:44,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 02:54:44,954 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490484954"}]},"ts":"1680490484954"} 2023-04-03 02:54:44,955 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 02:54:44,956 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 02:54:44,957 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03f67b48473b9761ac7a543789808247, UNASSIGN}] 2023-04-03 02:54:44,958 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03f67b48473b9761ac7a543789808247, UNASSIGN 2023-04-03 02:54:44,959 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=03f67b48473b9761ac7a543789808247, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:44,959 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490484959"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490484959"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490484959"}]},"ts":"1680490484959"} 2023-04-03 02:54:44,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:54:45,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 02:54:45,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:45,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 03f67b48473b9761ac7a543789808247, disabling compactions & flushes 2023-04-03 02:54:45,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:45,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:45,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. after waiting 0 ms 2023-04-03 02:54:45,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:45,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 03f67b48473b9761ac7a543789808247 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 02:54:45,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=A 2023-04-03 02:54:45,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:45,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=B 2023-04-03 02:54:45,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:45,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03f67b48473b9761ac7a543789808247, store=C 2023-04-03 02:54:45,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:45,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 02:54:45,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040368e7016438dc44dfa1d120d9237bcf25 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040368e7016438dc44dfa1d120d9237bcf25 2023-04-03 02:54:45,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 02:54:45,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=383, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/a9e3d197f25a4104beb2388bca96d94d 2023-04-03 02:54:46,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 02:54:46,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/a4b0c7b3c9694f7d92299c67c837af8b 2023-04-03 02:54:46,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/c41d00ddb3ba4b1e80faf206bdec2b21 2023-04-03 02:54:46,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/A/a9e3d197f25a4104beb2388bca96d94d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a9e3d197f25a4104beb2388bca96d94d 2023-04-03 02:54:46,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a9e3d197f25a4104beb2388bca96d94d, entries=100, sequenceid=383, filesize=18.6 K 2023-04-03 02:54:46,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/B/a4b0c7b3c9694f7d92299c67c837af8b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/a4b0c7b3c9694f7d92299c67c837af8b 2023-04-03 02:54:46,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/a4b0c7b3c9694f7d92299c67c837af8b, entries=100, sequenceid=383, filesize=9.5 K 2023-04-03 02:54:46,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/.tmp/C/c41d00ddb3ba4b1e80faf206bdec2b21 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/c41d00ddb3ba4b1e80faf206bdec2b21 2023-04-03 02:54:46,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/c41d00ddb3ba4b1e80faf206bdec2b21, entries=100, sequenceid=383, filesize=9.5 K 2023-04-03 02:54:46,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 03f67b48473b9761ac7a543789808247 in 1657ms, sequenceid=383, compaction requested=true 2023-04-03 02:54:46,788 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6f6c37a4e35142bf9536a6f7ddf6b918, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a9a2366a90bb4dc0b77c43410fed531f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/908aea553c8a468786e07e8c0c30b5cc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6df0ef07060344f8b45cb72521211123, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/1270d9a88a404304872bf863b0a9ba21, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/366f6d0c1c544dadbf65f0f30bd3d4a0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/21a5e10e6e6346db906531c1da975575, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a0d8bceb1ca04bcdb9a5e5c9f57c78fc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/0ac1d0d5e03a4aa6bf94a56ab46a72ce, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/93754b009f3b4a18b5a4cc532a116eb4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d1ccf912cca348e5b6276565b0c8925d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3be4fed221cf49fea053da487e12a37a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/7e1bbfc7bc424c9387d2e1ab69c63a11, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/b54aa1a9f8e24c3281bde438e0046237, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/99311577bd0e4055ba26994faf5e9aa2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/34c2175f20784e98afe5e422545945b4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/9ca002d54b4947c68d4ea238c192b652, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6d575df901004114ae114517781a306a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d3c68a9cdfdb444199bcaf3ddd8592dd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3e63a4e681a54768a0b8ca5c36245a75, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/c26ab44ae50e4abd9b7139d916b7eb3f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/00f296ef2fd64204bf837968926a57a8] to archive 2023-04-03 02:54:46,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:54:46,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6f6c37a4e35142bf9536a6f7ddf6b918 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6f6c37a4e35142bf9536a6f7ddf6b918 2023-04-03 02:54:46,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a9a2366a90bb4dc0b77c43410fed531f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a9a2366a90bb4dc0b77c43410fed531f 2023-04-03 02:54:46,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/908aea553c8a468786e07e8c0c30b5cc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/908aea553c8a468786e07e8c0c30b5cc 2023-04-03 02:54:46,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6df0ef07060344f8b45cb72521211123 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6df0ef07060344f8b45cb72521211123 2023-04-03 02:54:46,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/1270d9a88a404304872bf863b0a9ba21 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/1270d9a88a404304872bf863b0a9ba21 2023-04-03 02:54:46,799 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/366f6d0c1c544dadbf65f0f30bd3d4a0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/366f6d0c1c544dadbf65f0f30bd3d4a0 2023-04-03 02:54:46,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/21a5e10e6e6346db906531c1da975575 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/21a5e10e6e6346db906531c1da975575 2023-04-03 02:54:46,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a0d8bceb1ca04bcdb9a5e5c9f57c78fc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a0d8bceb1ca04bcdb9a5e5c9f57c78fc 2023-04-03 02:54:46,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/0ac1d0d5e03a4aa6bf94a56ab46a72ce to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/0ac1d0d5e03a4aa6bf94a56ab46a72ce 2023-04-03 02:54:46,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/93754b009f3b4a18b5a4cc532a116eb4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/93754b009f3b4a18b5a4cc532a116eb4 2023-04-03 02:54:46,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d1ccf912cca348e5b6276565b0c8925d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d1ccf912cca348e5b6276565b0c8925d 2023-04-03 02:54:46,808 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3be4fed221cf49fea053da487e12a37a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3be4fed221cf49fea053da487e12a37a 2023-04-03 02:54:46,810 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/7e1bbfc7bc424c9387d2e1ab69c63a11 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/7e1bbfc7bc424c9387d2e1ab69c63a11 2023-04-03 02:54:46,812 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/b54aa1a9f8e24c3281bde438e0046237 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/b54aa1a9f8e24c3281bde438e0046237 2023-04-03 02:54:46,813 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/99311577bd0e4055ba26994faf5e9aa2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/99311577bd0e4055ba26994faf5e9aa2 2023-04-03 02:54:46,814 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/34c2175f20784e98afe5e422545945b4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/34c2175f20784e98afe5e422545945b4 2023-04-03 02:54:46,816 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/9ca002d54b4947c68d4ea238c192b652 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/9ca002d54b4947c68d4ea238c192b652 2023-04-03 02:54:46,817 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6d575df901004114ae114517781a306a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/6d575df901004114ae114517781a306a 2023-04-03 02:54:46,819 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d3c68a9cdfdb444199bcaf3ddd8592dd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/d3c68a9cdfdb444199bcaf3ddd8592dd 2023-04-03 02:54:46,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3e63a4e681a54768a0b8ca5c36245a75 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/3e63a4e681a54768a0b8ca5c36245a75 2023-04-03 02:54:46,821 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/c26ab44ae50e4abd9b7139d916b7eb3f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/c26ab44ae50e4abd9b7139d916b7eb3f 2023-04-03 02:54:46,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/00f296ef2fd64204bf837968926a57a8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/00f296ef2fd64204bf837968926a57a8 2023-04-03 02:54:46,825 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/42991602ada2462995f145961aacfd74, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2ad9030320214a3e9957599a8ff1fb17, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/a54785d42c784dd29745e88e5578f53a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/cc5b7176632f40f2b0e2c402fb9d6fce, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/56ba68408e66421e80ff7a52dfa1dd17, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0eed3c3cc7534e79b8535bd9df3653e0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/8ddd6c89d3734f9c857465390761c6d2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0a6096f612e241a081f5d4ded287ad6a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/94cb3c513bce486d95f4066fd850c636, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/f4b1cb10fcc9454aaa89cd31f48ba74b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/62d77a9ac9b841b7b8529e06db9af5c2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/749f9735289f42b9b24cbe86bcae0298, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/9b4995537e7e4cf7b40708153bd95de1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d41f6cf4923848008f9c856e8bc5bf7c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/3ff328e0aba945c18b858595a83186dc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/781ad2a1e6ff418f827f10dd6157955e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/474a36a6b07e4084a9a231b969cce83c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d97304f9f9514691a27b65981d3a1ac7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/58616957961d48b9927031036d861b77, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/ef861fb77f0f43609829d6e76f91cdfe, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/83caedcd1e154707ad6c5cb237d7c2da, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2a3ff09bfb724288b027e57cd4a50e44] to archive 2023-04-03 02:54:46,826 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:54:46,828 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/42991602ada2462995f145961aacfd74 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/42991602ada2462995f145961aacfd74 2023-04-03 02:54:46,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2ad9030320214a3e9957599a8ff1fb17 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2ad9030320214a3e9957599a8ff1fb17 2023-04-03 02:54:46,830 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/a54785d42c784dd29745e88e5578f53a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/a54785d42c784dd29745e88e5578f53a 2023-04-03 02:54:46,831 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/cc5b7176632f40f2b0e2c402fb9d6fce to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/cc5b7176632f40f2b0e2c402fb9d6fce 2023-04-03 02:54:46,832 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/56ba68408e66421e80ff7a52dfa1dd17 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/56ba68408e66421e80ff7a52dfa1dd17 2023-04-03 02:54:46,833 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0eed3c3cc7534e79b8535bd9df3653e0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0eed3c3cc7534e79b8535bd9df3653e0 2023-04-03 02:54:46,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/8ddd6c89d3734f9c857465390761c6d2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/8ddd6c89d3734f9c857465390761c6d2 2023-04-03 02:54:46,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0a6096f612e241a081f5d4ded287ad6a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/0a6096f612e241a081f5d4ded287ad6a 2023-04-03 02:54:46,836 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/94cb3c513bce486d95f4066fd850c636 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/94cb3c513bce486d95f4066fd850c636 2023-04-03 02:54:46,836 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/f4b1cb10fcc9454aaa89cd31f48ba74b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/f4b1cb10fcc9454aaa89cd31f48ba74b 2023-04-03 02:54:46,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/62d77a9ac9b841b7b8529e06db9af5c2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/62d77a9ac9b841b7b8529e06db9af5c2 2023-04-03 02:54:46,838 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/749f9735289f42b9b24cbe86bcae0298 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/749f9735289f42b9b24cbe86bcae0298 2023-04-03 02:54:46,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/9b4995537e7e4cf7b40708153bd95de1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/9b4995537e7e4cf7b40708153bd95de1 2023-04-03 02:54:46,840 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d41f6cf4923848008f9c856e8bc5bf7c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d41f6cf4923848008f9c856e8bc5bf7c 2023-04-03 02:54:46,841 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/3ff328e0aba945c18b858595a83186dc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/3ff328e0aba945c18b858595a83186dc 2023-04-03 02:54:46,842 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/781ad2a1e6ff418f827f10dd6157955e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/781ad2a1e6ff418f827f10dd6157955e 2023-04-03 02:54:46,843 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/474a36a6b07e4084a9a231b969cce83c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/474a36a6b07e4084a9a231b969cce83c 2023-04-03 02:54:46,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d97304f9f9514691a27b65981d3a1ac7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/d97304f9f9514691a27b65981d3a1ac7 2023-04-03 02:54:46,845 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/58616957961d48b9927031036d861b77 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/58616957961d48b9927031036d861b77 2023-04-03 02:54:46,846 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/ef861fb77f0f43609829d6e76f91cdfe to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/ef861fb77f0f43609829d6e76f91cdfe 2023-04-03 02:54:46,847 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/83caedcd1e154707ad6c5cb237d7c2da to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/83caedcd1e154707ad6c5cb237d7c2da 2023-04-03 02:54:46,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2a3ff09bfb724288b027e57cd4a50e44 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/2a3ff09bfb724288b027e57cd4a50e44 2023-04-03 02:54:46,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/481c0c8f29744f1f9886a3da360c22c4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/48ae617802254fe398cff0cbf9e25f86, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/75d699a260734ae7a35f8595436a9776, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/60ab3823eb1d4dddb509dd6fcafdd047, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/76894e1b1ef8452ea6002595bbd23dd7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d0eef38a953d44749e3c1e993db33406, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/27f16b30f1014082999238a33d3d2c4e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/ff4aa26c11f54c8d96d2582c0d54ffad, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/0fc0716601424e70b88e27d06addf2b9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/add2905a180248d28995981e6f34dad7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/f810cc6ef021415dbe6df1f00dd47628, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/bfcc7e1c22c14b549262d3d2eadf3caa, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/9a9c9bb218c14f0e93cb57455bb493b2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/28723b37240046629e7a87e6a7339735, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/022b82aa567640ee9b287329fe665d99, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/b8e8a85b10b44f21b01ffb014054ba08, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/127643bc2aa6435d8a25ba08e872daac, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/e2d0c5fbe074404aa9b72e80819c4d83, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/3111399f347f474a895bfc05c4f2c1bc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/2f08127b26fc46f1a632d65be0360c41, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d8851715710747fea58807415c11537f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/1cef0b6443a74534951a35be65d2ebe2] to archive 2023-04-03 02:54:46,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:54:46,852 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/481c0c8f29744f1f9886a3da360c22c4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/481c0c8f29744f1f9886a3da360c22c4 2023-04-03 02:54:46,853 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/48ae617802254fe398cff0cbf9e25f86 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/48ae617802254fe398cff0cbf9e25f86 2023-04-03 02:54:46,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/75d699a260734ae7a35f8595436a9776 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/75d699a260734ae7a35f8595436a9776 2023-04-03 02:54:46,855 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/60ab3823eb1d4dddb509dd6fcafdd047 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/60ab3823eb1d4dddb509dd6fcafdd047 2023-04-03 02:54:46,856 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/76894e1b1ef8452ea6002595bbd23dd7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/76894e1b1ef8452ea6002595bbd23dd7 2023-04-03 02:54:46,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d0eef38a953d44749e3c1e993db33406 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d0eef38a953d44749e3c1e993db33406 2023-04-03 02:54:46,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/27f16b30f1014082999238a33d3d2c4e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/27f16b30f1014082999238a33d3d2c4e 2023-04-03 02:54:46,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/ff4aa26c11f54c8d96d2582c0d54ffad to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/ff4aa26c11f54c8d96d2582c0d54ffad 2023-04-03 02:54:46,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/0fc0716601424e70b88e27d06addf2b9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/0fc0716601424e70b88e27d06addf2b9 2023-04-03 02:54:46,861 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/add2905a180248d28995981e6f34dad7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/add2905a180248d28995981e6f34dad7 2023-04-03 02:54:46,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/f810cc6ef021415dbe6df1f00dd47628 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/f810cc6ef021415dbe6df1f00dd47628 2023-04-03 02:54:46,863 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/bfcc7e1c22c14b549262d3d2eadf3caa to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/bfcc7e1c22c14b549262d3d2eadf3caa 2023-04-03 02:54:46,865 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/9a9c9bb218c14f0e93cb57455bb493b2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/9a9c9bb218c14f0e93cb57455bb493b2 2023-04-03 02:54:46,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/28723b37240046629e7a87e6a7339735 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/28723b37240046629e7a87e6a7339735 2023-04-03 02:54:46,867 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/022b82aa567640ee9b287329fe665d99 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/022b82aa567640ee9b287329fe665d99 2023-04-03 02:54:46,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/b8e8a85b10b44f21b01ffb014054ba08 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/b8e8a85b10b44f21b01ffb014054ba08 2023-04-03 02:54:46,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/127643bc2aa6435d8a25ba08e872daac to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/127643bc2aa6435d8a25ba08e872daac 2023-04-03 02:54:46,869 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/e2d0c5fbe074404aa9b72e80819c4d83 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/e2d0c5fbe074404aa9b72e80819c4d83 2023-04-03 02:54:46,870 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/3111399f347f474a895bfc05c4f2c1bc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/3111399f347f474a895bfc05c4f2c1bc 2023-04-03 02:54:46,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/2f08127b26fc46f1a632d65be0360c41 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/2f08127b26fc46f1a632d65be0360c41 2023-04-03 02:54:46,872 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d8851715710747fea58807415c11537f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/d8851715710747fea58807415c11537f 2023-04-03 02:54:46,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/1cef0b6443a74534951a35be65d2ebe2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/1cef0b6443a74534951a35be65d2ebe2 2023-04-03 02:54:46,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/recovered.edits/386.seqid, newMaxSeqId=386, maxSeqId=4 2023-04-03 02:54:46,877 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247. 2023-04-03 02:54:46,877 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 03f67b48473b9761ac7a543789808247: 2023-04-03 02:54:46,879 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 03f67b48473b9761ac7a543789808247 2023-04-03 02:54:46,879 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=03f67b48473b9761ac7a543789808247, regionState=CLOSED 2023-04-03 02:54:46,879 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490486879"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490486879"}]},"ts":"1680490486879"} 2023-04-03 02:54:46,881 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-03 02:54:46,881 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 03f67b48473b9761ac7a543789808247, server=jenkins-hbase4.apache.org,43815,1680490421348 in 1.9200 sec 2023-04-03 02:54:46,882 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-03 02:54:46,882 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=03f67b48473b9761ac7a543789808247, UNASSIGN in 1.9240 sec 2023-04-03 02:54:46,883 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490486883"}]},"ts":"1680490486883"} 2023-04-03 02:54:46,884 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 02:54:46,886 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 02:54:46,887 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9370 sec 2023-04-03 02:54:47,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 02:54:47,058 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-03 02:54:47,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 02:54:47,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:47,061 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:47,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-03 02:54:47,061 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:47,065 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247 2023-04-03 02:54:47,067 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/recovered.edits] 2023-04-03 02:54:47,070 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/68d1670b0b7142f6bde7068717d6de29 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/68d1670b0b7142f6bde7068717d6de29 2023-04-03 02:54:47,071 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/841868d819894ff4ac93caa4d84f3e29 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/841868d819894ff4ac93caa4d84f3e29 2023-04-03 02:54:47,073 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a9e3d197f25a4104beb2388bca96d94d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/A/a9e3d197f25a4104beb2388bca96d94d 2023-04-03 02:54:47,075 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/5ca27ec83ee04081a61c0c65bd991d1b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/5ca27ec83ee04081a61c0c65bd991d1b 2023-04-03 02:54:47,077 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/a4b0c7b3c9694f7d92299c67c837af8b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/a4b0c7b3c9694f7d92299c67c837af8b 2023-04-03 02:54:47,078 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/ab1da32711714f498a4bcb513bb989dc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/B/ab1da32711714f498a4bcb513bb989dc 2023-04-03 02:54:47,081 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/c41d00ddb3ba4b1e80faf206bdec2b21 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/c41d00ddb3ba4b1e80faf206bdec2b21 2023-04-03 02:54:47,082 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/e178ae67a9bb4312a5fba09437cf8bdc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/e178ae67a9bb4312a5fba09437cf8bdc 2023-04-03 02:54:47,083 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/ef0b93af51b742ddae6c821b76f09b46 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/C/ef0b93af51b742ddae6c821b76f09b46 2023-04-03 02:54:47,085 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/recovered.edits/386.seqid to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247/recovered.edits/386.seqid 2023-04-03 02:54:47,086 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/03f67b48473b9761ac7a543789808247 2023-04-03 02:54:47,086 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:54:47,086 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 02:54:47,087 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-03 02:54:47,090 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030bf9938d731c4379bf5d7d693041d037 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030bf9938d731c4379bf5d7d693041d037 2023-04-03 02:54:47,091 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032a4f8c020fbd4308bb1432d6f5a5af2c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032a4f8c020fbd4308bb1432d6f5a5af2c 2023-04-03 02:54:47,093 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032a7ee6d0d3e24272933c2e3c5775d1e5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032a7ee6d0d3e24272933c2e3c5775d1e5 2023-04-03 02:54:47,093 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403334cf9054a734a459e3c12dfbd19a224 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403334cf9054a734a459e3c12dfbd19a224 2023-04-03 02:54:47,094 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040333dd610f7ddf4ce1971e190ede7bf1c8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040333dd610f7ddf4ce1971e190ede7bf1c8 2023-04-03 02:54:47,095 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040341e2d6860f034c3483440dd3e1c0e8ad to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040341e2d6860f034c3483440dd3e1c0e8ad 2023-04-03 02:54:47,096 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403436d7fd6a0b341e4a3f4135ef77d62f0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403436d7fd6a0b341e4a3f4135ef77d62f0 2023-04-03 02:54:47,097 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403565bfd44a001439680e1e23a48931c97 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403565bfd44a001439680e1e23a48931c97 2023-04-03 02:54:47,098 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040368e7016438dc44dfa1d120d9237bcf25 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040368e7016438dc44dfa1d120d9237bcf25 2023-04-03 02:54:47,099 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403810c3b4b7c3347dab09a6619b8a1f3b3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403810c3b4b7c3347dab09a6619b8a1f3b3 2023-04-03 02:54:47,100 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038731403200f44fd69c2e2ffa9ae6552c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038731403200f44fd69c2e2ffa9ae6552c 2023-04-03 02:54:47,101 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a3187788a622416f890c6e5eb56048c2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a3187788a622416f890c6e5eb56048c2 2023-04-03 02:54:47,102 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a45269d1a32c4ce19cc27e30fdd5701d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a45269d1a32c4ce19cc27e30fdd5701d 2023-04-03 02:54:47,104 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b35a6953ff6540ff831d456a69cdee75 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b35a6953ff6540ff831d456a69cdee75 2023-04-03 02:54:47,105 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d74b5245784942f69c17c0b7d07451a7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d74b5245784942f69c17c0b7d07451a7 2023-04-03 02:54:47,106 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d753a7bbdf9e42e69882b6e19ec0b73e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d753a7bbdf9e42e69882b6e19ec0b73e 2023-04-03 02:54:47,107 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e87987cbe34c4b6195629ac23c204ab4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e87987cbe34c4b6195629ac23c204ab4 2023-04-03 02:54:47,108 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403eadbbb5286524cadbdc36632fef3bdeb to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403eadbbb5286524cadbdc36632fef3bdeb 2023-04-03 02:54:47,109 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f631696fbf144e9e8e70f7063d617e4f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f631696fbf144e9e8e70f7063d617e4f 2023-04-03 02:54:47,110 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fceb2823c79747abaca81f82bb504e8e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fceb2823c79747abaca81f82bb504e8e 2023-04-03 02:54:47,111 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 02:54:47,113 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:47,115 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 02:54:47,117 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 02:54:47,118 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:47,118 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 02:54:47,118 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680490487118"}]},"ts":"9223372036854775807"} 2023-04-03 02:54:47,120 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 02:54:47,120 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 03f67b48473b9761ac7a543789808247, NAME => 'TestAcidGuarantees,,1680490452245.03f67b48473b9761ac7a543789808247.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 02:54:47,120 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 02:54:47,120 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680490487120"}]},"ts":"9223372036854775807"} 2023-04-03 02:54:47,121 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 02:54:47,123 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:47,123 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-04-03 02:54:47,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-03 02:54:47,162 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-03 02:54:47,178 INFO [Listener at localhost/46853] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 233), OpenFileDescriptor=507 (was 551), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=396 (was 455), ProcessCount=170 (was 170), AvailableMemoryMB=3590 (was 3765) 2023-04-03 02:54:47,189 INFO [Listener at localhost/46853] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=507, MaxFileDescriptor=60000, SystemLoadAverage=396, ProcessCount=170, AvailableMemoryMB=3589 2023-04-03 02:54:47,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 02:54:47,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 02:54:47,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 02:54:47,194 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 02:54:47,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-03 02:54:47,194 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 02:54:47,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 02:54:47,196 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:47,197 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369 empty. 2023-04-03 02:54:47,197 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:47,197 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:54:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 02:54:47,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 02:54:47,605 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 02:54:47,607 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ca59c59c9ebb324c8ea1a6deda6d6369, NAME => 'TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp 2023-04-03 02:54:47,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 02:54:48,012 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:54:48,013 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ca59c59c9ebb324c8ea1a6deda6d6369, disabling compactions & flushes 2023-04-03 02:54:48,013 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:48,013 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:48,013 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. after waiting 0 ms 2023-04-03 02:54:48,013 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:48,013 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:48,013 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:48,015 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 02:54:48,016 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490488015"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490488015"}]},"ts":"1680490488015"} 2023-04-03 02:54:48,017 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 02:54:48,017 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 02:54:48,017 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490488017"}]},"ts":"1680490488017"} 2023-04-03 02:54:48,018 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 02:54:48,021 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca59c59c9ebb324c8ea1a6deda6d6369, ASSIGN}] 2023-04-03 02:54:48,023 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca59c59c9ebb324c8ea1a6deda6d6369, ASSIGN 2023-04-03 02:54:48,023 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca59c59c9ebb324c8ea1a6deda6d6369, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43815,1680490421348; forceNewPlan=false, retain=false 2023-04-03 02:54:48,174 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=ca59c59c9ebb324c8ea1a6deda6d6369, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:48,174 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490488174"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490488174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490488174"}]},"ts":"1680490488174"} 2023-04-03 02:54:48,176 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:54:48,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 02:54:48,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:48,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ca59c59c9ebb324c8ea1a6deda6d6369, NAME => 'TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:54:48,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:48,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:54:48,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:48,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:48,332 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:48,333 DEBUG [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A 2023-04-03 02:54:48,333 DEBUG [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A 2023-04-03 02:54:48,334 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:54:48,334 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca59c59c9ebb324c8ea1a6deda6d6369 columnFamilyName A 2023-04-03 02:54:48,335 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] regionserver.HStore(310): Store=ca59c59c9ebb324c8ea1a6deda6d6369/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:54:48,335 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:48,336 DEBUG [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B 2023-04-03 02:54:48,336 DEBUG [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B 2023-04-03 02:54:48,336 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:54:48,336 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca59c59c9ebb324c8ea1a6deda6d6369 columnFamilyName B 2023-04-03 02:54:48,337 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] regionserver.HStore(310): Store=ca59c59c9ebb324c8ea1a6deda6d6369/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:54:48,337 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:48,338 DEBUG [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C 2023-04-03 02:54:48,338 DEBUG [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C 2023-04-03 02:54:48,338 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:54:48,338 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca59c59c9ebb324c8ea1a6deda6d6369 columnFamilyName C 2023-04-03 02:54:48,339 INFO [StoreOpener-ca59c59c9ebb324c8ea1a6deda6d6369-1] regionserver.HStore(310): Store=ca59c59c9ebb324c8ea1a6deda6d6369/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:54:48,339 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:48,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:48,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:48,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 02:54:48,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:48,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 02:54:48,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ca59c59c9ebb324c8ea1a6deda6d6369; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70706131, jitterRate=0.05360345542430878}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 02:54:48,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:48,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., pid=30, masterSystemTime=1680490488327 2023-04-03 02:54:48,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:48,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:48,346 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=ca59c59c9ebb324c8ea1a6deda6d6369, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:48,346 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490488346"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680490488346"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680490488346"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680490488346"}]},"ts":"1680490488346"} 2023-04-03 02:54:48,348 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-03 02:54:48,348 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 in 171 msec 2023-04-03 02:54:48,350 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-03 02:54:48,350 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca59c59c9ebb324c8ea1a6deda6d6369, ASSIGN in 327 msec 2023-04-03 02:54:48,350 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 02:54:48,350 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490488350"}]},"ts":"1680490488350"} 2023-04-03 02:54:48,351 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 02:54:48,353 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 02:54:48,354 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-03 02:54:49,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 02:54:49,299 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-03 02:54:49,300 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x7ed1dde9 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:49,304 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262f4570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:49,306 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:49,307 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:49,308 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 02:54:49,310 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53110, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 02:54:49,311 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:49,315 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:49,315 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:49,321 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:49,322 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:49,325 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:49,325 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:49,329 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:49,329 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x7810a231 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:49,332 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@104dbf4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:49,332 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x753156b2 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:49,339 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@87b3095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:49,339 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:49,342 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:49,343 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:49,346 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:49,346 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:49,349 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:49,349 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x2de064cb to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:54:49,353 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f8ff99f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:54:49,357 DEBUG [hconnection-0x3ca17b7b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:49,357 DEBUG [hconnection-0x5bd57ed3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:49,358 DEBUG [hconnection-0x6f8ab739-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:49,358 DEBUG [hconnection-0x29fdf911-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:49,358 DEBUG [hconnection-0x1ccd5db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:49,359 DEBUG [hconnection-0x63047db3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:49,359 DEBUG [hconnection-0x2850d49f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:49,360 DEBUG [hconnection-0x2966c99d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:49,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:49,360 DEBUG [hconnection-0x6667e8b6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:49,361 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33714, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:49,361 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33738, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:49,362 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:49,362 DEBUG [hconnection-0x5b6c3c6a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:54:49,373 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:49,373 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33694, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:49,374 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33720, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:49,374 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33750, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:49,374 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33786, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:49,373 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33732, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:49,376 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:54:49,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:49,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:49,378 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:49,378 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:49,378 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:49,378 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:49,381 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,381 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:49,382 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:49,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:49,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:49,388 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 02:54:49,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:49,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:49,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:49,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:49,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:49,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:49,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:49,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:49,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:49,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:49,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:49,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:49,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:49,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:49,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:49,398 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,398 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733da5[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:49,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:49,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,402 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:49,402 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:49,402 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,402 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:49,404 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:49,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:49,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:49,407 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:49,408 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:49,408 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:49,408 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:49,408 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:49,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,410 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:49,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:49,410 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,412 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,415 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,415 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,415 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,415 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:49,415 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:49,415 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,415 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,415 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:49,415 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:49,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:49,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:49,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:49,415 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:49,416 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:49,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:49,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490549409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490549413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490549413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490549414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490549415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/2c286d20d6c24bcfa7da8a88e4f1876b 2023-04-03 02:54:49,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/d806662321f843bba0120d8e025e62d9 2023-04-03 02:54:49,515 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/1bb9a9b46dae4e578aa9a4c77fa73f30 2023-04-03 02:54:49,516 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:49,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:49,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490549517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490549518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490549518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:49,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:49,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:49,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490549519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,521 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:49,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/2c286d20d6c24bcfa7da8a88e4f1876b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2c286d20d6c24bcfa7da8a88e4f1876b 2023-04-03 02:54:49,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490549519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,521 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:49,522 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:49,522 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:49,522 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,522 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,524 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:49,524 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:49,525 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,525 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:49,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:49,528 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:49,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2c286d20d6c24bcfa7da8a88e4f1876b, entries=150, sequenceid=13, filesize=11.6 K 2023-04-03 02:54:49,528 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:49,528 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:49,528 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:49,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:49,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/d806662321f843bba0120d8e025e62d9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d806662321f843bba0120d8e025e62d9 2023-04-03 02:54:49,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d806662321f843bba0120d8e025e62d9, entries=150, sequenceid=13, filesize=11.6 K 2023-04-03 02:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:49,536 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:49,536 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442ea0e6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:49,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/1bb9a9b46dae4e578aa9a4c77fa73f30 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1bb9a9b46dae4e578aa9a4c77fa73f30 2023-04-03 02:54:49,541 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,541 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,541 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:49,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,542 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:49,542 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:49,542 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:49,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1bb9a9b46dae4e578aa9a4c77fa73f30, entries=150, sequenceid=13, filesize=11.6 K 2023-04-03 02:54:49,546 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:49,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:49,546 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ca59c59c9ebb324c8ea1a6deda6d6369 in 158ms, sequenceid=13, compaction requested=false 2023-04-03 02:54:49,546 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:49,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:49,550 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:49,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:49,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:49,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:49,550 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:49,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:49,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,555 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:49,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:49,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:49,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,563 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,563 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,563 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,563 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,563 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,563 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:49,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:49,563 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:49,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:49,564 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:49,564 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:49,664 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:49,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:49,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:49,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:49,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:49,667 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:49,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:49,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:49,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:49,668 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,668 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,669 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:49,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:49,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:49,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:49,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:49,671 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:49,671 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:49,671 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:49,671 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:49,671 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:49,671 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:49,671 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:49,671 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 02:54:49,671 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:49,671 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:49,671 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:49,671 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:49,671 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:49,671 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:49,701 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/c176ae78cb1f4fe0bc45fa78dfd76a67 2023-04-03 02:54:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:49,722 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:49,727 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/4ebd707b4e41478399da8a1921d66434 2023-04-03 02:54:49,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490549726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490549727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490549727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490549728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490549731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,754 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/fadc9e7128dd41ce97e571a469974cea 2023-04-03 02:54:49,758 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/c176ae78cb1f4fe0bc45fa78dfd76a67 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c176ae78cb1f4fe0bc45fa78dfd76a67 2023-04-03 02:54:49,763 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c176ae78cb1f4fe0bc45fa78dfd76a67, entries=150, sequenceid=33, filesize=11.6 K 2023-04-03 02:54:49,763 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/4ebd707b4e41478399da8a1921d66434 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4ebd707b4e41478399da8a1921d66434 2023-04-03 02:54:49,767 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4ebd707b4e41478399da8a1921d66434, entries=150, sequenceid=33, filesize=11.6 K 2023-04-03 02:54:49,770 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/fadc9e7128dd41ce97e571a469974cea as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/fadc9e7128dd41ce97e571a469974cea 2023-04-03 02:54:49,774 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/fadc9e7128dd41ce97e571a469974cea, entries=150, sequenceid=33, filesize=11.6 K 2023-04-03 02:54:49,775 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ca59c59c9ebb324c8ea1a6deda6d6369 in 104ms, sequenceid=33, compaction requested=false 2023-04-03 02:54:49,775 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:49,775 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:49,777 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d35f83[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:49,779 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:49,780 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,782 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:49,782 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,782 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:49,784 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:49,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:49,787 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:49,787 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:49,787 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:49,787 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:49,787 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:49,789 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,789 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,789 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,790 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,791 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:49,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:49,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:49,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:49,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:49,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:49,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:49,794 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:49,794 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:49,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:49,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:49,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 02:54:49,835 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:49,835 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:49,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:49,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:49,836 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:49,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:49,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490549849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490549850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490549851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490549851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490549852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,894 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:49,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:49,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:49,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:49,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:49,898 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:49,898 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:49,898 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:49,898 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:49,898 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,898 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,900 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:49,900 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:49,901 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,901 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:49,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:49,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:49,902 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:49,902 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:49,902 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:49,902 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:49,902 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:49,902 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:49,902 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:49,902 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:49,902 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:49,902 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:49,902 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:49,902 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:49,904 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,904 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,904 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,904 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:49,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b4f8510[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:49,906 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:49,906 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,907 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:49,907 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,907 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:49,909 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:49,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:49,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:49,912 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:49,912 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:49,912 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:49,912 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:49,912 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:49,917 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,917 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:49,917 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:49,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:49,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:49,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:49,925 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:49,925 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:49,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:49,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:49,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:49,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:49,925 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,925 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:49,925 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:49,925 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:49,925 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,926 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:49,926 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,926 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:49,926 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:49,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:49,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490549953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490549954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490549955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490549955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:49,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:49,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490549955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,026 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:50,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:50,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:50,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:50,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:50,029 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:50,029 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,029 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:50,029 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:50,030 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,030 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,031 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,031 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,031 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,032 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:50,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:50,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:50,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:50,033 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,033 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:50,033 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:50,033 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:50,034 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:50,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524817c9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:50,037 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:50,037 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,039 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,039 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:50,039 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:50,039 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:50,039 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,039 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:50,040 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,041 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:50,041 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,041 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:50,043 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:50,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:50,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:50,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:50,043 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:50,045 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,045 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,045 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,045 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,052 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,052 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,052 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,052 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,052 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,052 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,052 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,052 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:50,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:50,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,052 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,052 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,052 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:50,053 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:50,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,153 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:50,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:50,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:50,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:50,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:50,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490550155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,157 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:50,157 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,157 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:50,157 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:50,158 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,158 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490550157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,159 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,160 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490550158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490550158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490550159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:50,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:50,161 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:50,161 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,161 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:50,163 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,163 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:50,164 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:50,164 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:50,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:50,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6350a4f6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:50,168 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:50,168 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:50,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:50,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:50,170 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,170 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:50,171 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,171 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:50,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:50,174 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:50,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:50,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:50,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:50,174 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:50,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,177 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,181 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:50,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:50,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:50,184 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,184 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:50,185 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:50,247 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/00b68aa8332d413e8c25dbf7d508a546 2023-04-03 02:54:50,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/7dbbda6e5d6e428f95359d3be9978309 2023-04-03 02:54:50,285 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:50,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:50,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:50,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:50,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:50,304 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:50,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:50,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:50,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,306 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,307 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,307 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:50,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:50,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:50,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:50,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:50,312 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:50,312 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:50,312 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/2ac6fcd768e64fd6874d94fe1acf1a1a 2023-04-03 02:54:50,312 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,312 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:50,313 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:50,313 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,313 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:50,313 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:50,313 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,313 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:50,313 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:50,316 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:50,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:50,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a56c058[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:50,319 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:50,319 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/00b68aa8332d413e8c25dbf7d508a546 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/00b68aa8332d413e8c25dbf7d508a546 2023-04-03 02:54:50,320 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:50,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:50,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:50,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:50,321 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,321 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:50,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:50,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:50,324 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:50,324 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:50,324 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:50,324 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:50,324 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:50,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/00b68aa8332d413e8c25dbf7d508a546, entries=150, sequenceid=48, filesize=11.6 K 2023-04-03 02:54:50,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/7dbbda6e5d6e428f95359d3be9978309 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7dbbda6e5d6e428f95359d3be9978309 2023-04-03 02:54:50,326 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,326 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,326 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,327 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,329 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7dbbda6e5d6e428f95359d3be9978309, entries=150, sequenceid=48, filesize=11.6 K 2023-04-03 02:54:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/2ac6fcd768e64fd6874d94fe1acf1a1a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/2ac6fcd768e64fd6874d94fe1acf1a1a 2023-04-03 02:54:50,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:50,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:50,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,333 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,333 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,333 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,333 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,333 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:50,333 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:50,332 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:50,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/2ac6fcd768e64fd6874d94fe1acf1a1a, entries=150, sequenceid=48, filesize=11.6 K 2023-04-03 02:54:50,337 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for ca59c59c9ebb324c8ea1a6deda6d6369 in 503ms, sequenceid=48, compaction requested=true 2023-04-03 02:54:50,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:50,338 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:50,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:50,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:50,338 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:50,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:50,339 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:50,339 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:54:50,339 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,339 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2c286d20d6c24bcfa7da8a88e4f1876b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c176ae78cb1f4fe0bc45fa78dfd76a67, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/00b68aa8332d413e8c25dbf7d508a546] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=34.8 K 2023-04-03 02:54:50,340 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 2c286d20d6c24bcfa7da8a88e4f1876b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680490489382 2023-04-03 02:54:50,340 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:50,340 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:54:50,340 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c176ae78cb1f4fe0bc45fa78dfd76a67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680490489406 2023-04-03 02:54:50,340 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,340 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d806662321f843bba0120d8e025e62d9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4ebd707b4e41478399da8a1921d66434, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7dbbda6e5d6e428f95359d3be9978309] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=34.8 K 2023-04-03 02:54:50,341 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d806662321f843bba0120d8e025e62d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680490489382 2023-04-03 02:54:50,341 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 00b68aa8332d413e8c25dbf7d508a546, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680490489726 2023-04-03 02:54:50,341 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 4ebd707b4e41478399da8a1921d66434, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680490489406 2023-04-03 02:54:50,342 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7dbbda6e5d6e428f95359d3be9978309, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680490489726 2023-04-03 02:54:50,350 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#199 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:50,350 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#198 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:50,386 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/b48e086d32294bf3ba0de888e9f1d469 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b48e086d32294bf3ba0de888e9f1d469 2023-04-03 02:54:50,389 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/09c9e5a08e214ecf87edddb5174e529d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/09c9e5a08e214ecf87edddb5174e529d 2023-04-03 02:54:50,391 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into b48e086d32294bf3ba0de888e9f1d469(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:50,391 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:50,391 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=13, startTime=1680490490338; duration=0sec 2023-04-03 02:54:50,391 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:50,392 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:50,393 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:50,394 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:54:50,394 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,394 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1bb9a9b46dae4e578aa9a4c77fa73f30, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/fadc9e7128dd41ce97e571a469974cea, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/2ac6fcd768e64fd6874d94fe1acf1a1a] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=34.8 K 2023-04-03 02:54:50,394 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 1bb9a9b46dae4e578aa9a4c77fa73f30, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680490489382 2023-04-03 02:54:50,394 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting fadc9e7128dd41ce97e571a469974cea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680490489406 2023-04-03 02:54:50,396 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into 09c9e5a08e214ecf87edddb5174e529d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:50,396 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:50,396 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=13, startTime=1680490490338; duration=0sec 2023-04-03 02:54:50,396 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:50,396 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 2ac6fcd768e64fd6874d94fe1acf1a1a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680490489726 2023-04-03 02:54:50,403 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:50,413 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/6030c50e962a4f96be0167dd704e9d9b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/6030c50e962a4f96be0167dd704e9d9b 2023-04-03 02:54:50,418 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into 6030c50e962a4f96be0167dd704e9d9b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:50,418 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:50,418 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=13, startTime=1680490490338; duration=0sec 2023-04-03 02:54:50,418 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:50,433 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:50,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:50,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:50,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:50,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:50,436 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:50,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:50,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:50,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,437 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,438 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,438 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,438 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,438 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:50,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:50,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:50,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:50,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:50,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:50,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:50,439 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:50,440 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 02:54:50,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:50,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:50,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:50,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:50,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:50,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:50,450 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/016780c8a84d4b2fa6a337a66d0669a9 2023-04-03 02:54:50,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:50,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:50,467 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/6d073555a4414f3095c5f93d0664f644 2023-04-03 02:54:50,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490550468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490550469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490550470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490550471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490550471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,511 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/e2e0f9750ccb435f9083f7e5805b12c9 2023-04-03 02:54:50,517 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/016780c8a84d4b2fa6a337a66d0669a9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/016780c8a84d4b2fa6a337a66d0669a9 2023-04-03 02:54:50,525 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/016780c8a84d4b2fa6a337a66d0669a9, entries=150, sequenceid=69, filesize=11.6 K 2023-04-03 02:54:50,526 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/6d073555a4414f3095c5f93d0664f644 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6d073555a4414f3095c5f93d0664f644 2023-04-03 02:54:50,537 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6d073555a4414f3095c5f93d0664f644, entries=150, sequenceid=69, filesize=11.6 K 2023-04-03 02:54:50,538 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/e2e0f9750ccb435f9083f7e5805b12c9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e2e0f9750ccb435f9083f7e5805b12c9 2023-04-03 02:54:50,543 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e2e0f9750ccb435f9083f7e5805b12c9, entries=150, sequenceid=69, filesize=11.6 K 2023-04-03 02:54:50,543 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ca59c59c9ebb324c8ea1a6deda6d6369 in 103ms, sequenceid=69, compaction requested=false 2023-04-03 02:54:50,544 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:50,544 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:50,545 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,546 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:50,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d86ab07[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:50,549 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:50,549 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,550 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:50,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:50,553 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:50,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:50,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:50,556 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:50,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:50,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:50,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:50,556 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:50,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,564 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:50,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:50,565 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,565 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:50,565 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:50,565 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:50,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:50,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 02:54:50,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:50,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:50,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:50,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:50,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:50,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:50,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490550584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490550585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490550587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490550588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490550589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,665 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:50,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:50,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:50,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:50,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:50,668 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:50,668 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,668 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:50,668 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:50,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,671 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:50,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:50,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:50,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:50,673 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:50,673 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,673 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:50,673 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:50,673 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:50,673 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,673 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:50,673 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:50,673 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,673 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:50,673 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:50,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:50,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:50,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,675 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,675 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,675 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:50,677 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:50,677 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f621d21[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:50,678 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,678 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:50,678 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,678 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:50,678 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:50,678 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:50,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,680 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:50,680 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,680 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:50,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:50,682 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:50,682 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:50,682 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:50,682 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:50,682 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:50,684 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,684 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,685 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,686 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,688 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,688 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:50,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:50,688 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,689 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:50,689 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:50,688 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:50,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,689 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,689 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490550690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490550690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490550691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490550692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490550692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,789 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:50,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:50,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:50,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:50,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:50,792 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:50,792 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,792 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:50,792 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:50,793 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,793 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:50,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:50,795 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:50,796 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,796 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:50,796 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:50,796 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:50,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:50,800 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:50,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9f3632f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:50,800 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,802 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:50,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:50,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:50,802 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,802 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:50,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,803 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,803 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,803 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:50,806 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:50,806 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:50,806 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:50,806 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:50,806 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:50,807 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,807 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,808 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,808 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,809 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,812 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,812 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:50,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:50,812 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,812 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,812 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,812 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,812 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:50,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:50,812 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,812 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:50,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,812 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490550893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490550893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490550894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490550894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:50,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490550895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,912 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:50,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:50,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:50,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:50,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:50,915 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:50,915 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,915 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:50,915 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:50,916 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,916 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,918 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,918 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,918 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,918 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:50,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:50,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:50,919 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:50,919 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:50,920 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,920 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:50,920 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:50,920 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:50,922 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b3649[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:50,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:50,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,924 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,924 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,924 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:50,926 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:50,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:50,928 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:50,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:50,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:50,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:50,928 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:50,930 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,931 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,931 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:50,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,931 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,932 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:50,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:50,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:50,934 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:50,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:50,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:50,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:50,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:50,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:50,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:50,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:50,935 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:50,935 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:50,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:50,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/8389be08afdf450aa8c495eab0d6051d 2023-04-03 02:54:50,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/050d7d353a054a35b51d2af279b14aa4 2023-04-03 02:54:51,016 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/41023184e1eb4148b71d01851c798aac 2023-04-03 02:54:51,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/8389be08afdf450aa8c495eab0d6051d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8389be08afdf450aa8c495eab0d6051d 2023-04-03 02:54:51,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8389be08afdf450aa8c495eab0d6051d, entries=200, sequenceid=85, filesize=13.9 K 2023-04-03 02:54:51,026 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/050d7d353a054a35b51d2af279b14aa4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/050d7d353a054a35b51d2af279b14aa4 2023-04-03 02:54:51,029 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/050d7d353a054a35b51d2af279b14aa4, entries=150, sequenceid=85, filesize=11.6 K 2023-04-03 02:54:51,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/41023184e1eb4148b71d01851c798aac as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/41023184e1eb4148b71d01851c798aac 2023-04-03 02:54:51,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/41023184e1eb4148b71d01851c798aac, entries=150, sequenceid=85, filesize=11.6 K 2023-04-03 02:54:51,035 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for ca59c59c9ebb324c8ea1a6deda6d6369 in 460ms, sequenceid=85, compaction requested=true 2023-04-03 02:54:51,035 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:51,035 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:51,035 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:51,035 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:51,035 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:51,035 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:51,035 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:51,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:51,036 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:51,036 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:54:51,036 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:51,036 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:51,036 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/09c9e5a08e214ecf87edddb5174e529d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/016780c8a84d4b2fa6a337a66d0669a9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8389be08afdf450aa8c495eab0d6051d] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=37.2 K 2023-04-03 02:54:51,036 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:54:51,036 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:51,036 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b48e086d32294bf3ba0de888e9f1d469, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6d073555a4414f3095c5f93d0664f644, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/050d7d353a054a35b51d2af279b14aa4] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=34.9 K 2023-04-03 02:54:51,036 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 09c9e5a08e214ecf87edddb5174e529d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680490489726 2023-04-03 02:54:51,037 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b48e086d32294bf3ba0de888e9f1d469, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680490489726 2023-04-03 02:54:51,037 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 016780c8a84d4b2fa6a337a66d0669a9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680490489844 2023-04-03 02:54:51,037 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 6d073555a4414f3095c5f93d0664f644, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680490489844 2023-04-03 02:54:51,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:51,037 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 8389be08afdf450aa8c495eab0d6051d, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680490490469 2023-04-03 02:54:51,038 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 050d7d353a054a35b51d2af279b14aa4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680490490469 2023-04-03 02:54:51,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:51,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:51,039 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:51,039 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:51,039 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:51,039 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:51,039 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:51,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:51,046 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,046 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:51,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:51,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:51,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:51,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:51,049 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:51,053 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:51,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:51,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:51,072 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:51,072 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:51,072 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 02:54:51,072 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:51,072 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:51,072 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:51,072 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:51,072 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:51,073 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:51,082 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/dc96198d5d3f4b31801f402de5255d18 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/dc96198d5d3f4b31801f402de5255d18 2023-04-03 02:54:51,090 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into dc96198d5d3f4b31801f402de5255d18(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:51,090 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:51,090 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=13, startTime=1680490491035; duration=0sec 2023-04-03 02:54:51,091 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:51,091 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:51,092 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:51,092 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:54:51,092 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:51,092 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/6030c50e962a4f96be0167dd704e9d9b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e2e0f9750ccb435f9083f7e5805b12c9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/41023184e1eb4148b71d01851c798aac] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=34.9 K 2023-04-03 02:54:51,092 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6030c50e962a4f96be0167dd704e9d9b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680490489726 2023-04-03 02:54:51,094 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e2e0f9750ccb435f9083f7e5805b12c9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680490489844 2023-04-03 02:54:51,094 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/60c06a7dbe2d45f2b37d35cb58d46278 2023-04-03 02:54:51,095 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 41023184e1eb4148b71d01851c798aac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680490490469 2023-04-03 02:54:51,097 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/494aca889df8480f9fed66526ff19cc4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/494aca889df8480f9fed66526ff19cc4 2023-04-03 02:54:51,104 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into 494aca889df8480f9fed66526ff19cc4(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:51,104 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:51,104 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=13, startTime=1680490491035; duration=0sec 2023-04-03 02:54:51,104 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:51,105 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#211 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:51,119 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/c9845fd868944621960d01401c265dbe 2023-04-03 02:54:51,129 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/5a7fcbab4c434806990ce75ff7408ddd 2023-04-03 02:54:51,133 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/60c06a7dbe2d45f2b37d35cb58d46278 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60c06a7dbe2d45f2b37d35cb58d46278 2023-04-03 02:54:51,137 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60c06a7dbe2d45f2b37d35cb58d46278, entries=150, sequenceid=101, filesize=11.6 K 2023-04-03 02:54:51,138 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/c9845fd868944621960d01401c265dbe as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c9845fd868944621960d01401c265dbe 2023-04-03 02:54:51,141 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c9845fd868944621960d01401c265dbe, entries=150, sequenceid=101, filesize=11.6 K 2023-04-03 02:54:51,142 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/5a7fcbab4c434806990ce75ff7408ddd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/5a7fcbab4c434806990ce75ff7408ddd 2023-04-03 02:54:51,146 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/5a7fcbab4c434806990ce75ff7408ddd, entries=150, sequenceid=101, filesize=11.6 K 2023-04-03 02:54:51,146 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for ca59c59c9ebb324c8ea1a6deda6d6369 in 74ms, sequenceid=101, compaction requested=false 2023-04-03 02:54:51,146 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:51,147 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:51,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:51,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:51,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:51,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:51,148 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:51,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:51,149 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:51,149 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:51,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:51,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d4d6576[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:51,151 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:51,151 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:51,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:51,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:51,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:51,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:51,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:51,152 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,152 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:51,153 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:51,154 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:51,154 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:51,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:51,156 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:51,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:51,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:51,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:51,157 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:51,159 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:51,159 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:51,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:51,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:51,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:51,160 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:51,162 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:51,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:51,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:51,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:51,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:51,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:51,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:51,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:51,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:51,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,165 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:51,165 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:51,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:51,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:51,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:51,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:51,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:51,201 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:54:51,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:51,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:51,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:51,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:51,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:51,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:51,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/99245e0a51ee4b4480b43449161285d1 2023-04-03 02:54:51,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490491210, max timestamp=1680490491216Num uniques 859; 2023-04-03 02:54:51,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490491210, max timestamp=1680490491216Num uniques 859; 2023-04-03 02:54:51,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:54:51,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:54:51,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490491210, max timestamp=1680490491216Num uniques 859; 2023-04-03 02:54:51,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:54:51,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/434d0e6cb89f455d960dbbb28b0cda74 2023-04-03 02:54:51,232 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/8f90f1049d4c4d36a2404ea8e5bc9522 2023-04-03 02:54:51,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490551232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490551232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490551232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490551233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490551234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/99245e0a51ee4b4480b43449161285d1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/99245e0a51ee4b4480b43449161285d1 2023-04-03 02:54:51,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/99245e0a51ee4b4480b43449161285d1, entries=150, sequenceid=113, filesize=11.6 K 2023-04-03 02:54:51,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/434d0e6cb89f455d960dbbb28b0cda74 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/434d0e6cb89f455d960dbbb28b0cda74 2023-04-03 02:54:51,245 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/434d0e6cb89f455d960dbbb28b0cda74, entries=100, sequenceid=113, filesize=9.3 K 2023-04-03 02:54:51,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/8f90f1049d4c4d36a2404ea8e5bc9522 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8f90f1049d4c4d36a2404ea8e5bc9522 2023-04-03 02:54:51,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8f90f1049d4c4d36a2404ea8e5bc9522, entries=100, sequenceid=113, filesize=9.3 K 2023-04-03 02:54:51,250 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ca59c59c9ebb324c8ea1a6deda6d6369 in 49ms, sequenceid=113, compaction requested=true 2023-04-03 02:54:51,250 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:51,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:51,250 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:51,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:51,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:54:51,251 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35860 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:51,251 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:54:51,251 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:51,252 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/dc96198d5d3f4b31801f402de5255d18, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60c06a7dbe2d45f2b37d35cb58d46278, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/99245e0a51ee4b4480b43449161285d1] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=35.0 K 2023-04-03 02:54:51,252 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting dc96198d5d3f4b31801f402de5255d18, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680490490469 2023-04-03 02:54:51,252 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 60c06a7dbe2d45f2b37d35cb58d46278, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680490490581 2023-04-03 02:54:51,253 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 99245e0a51ee4b4480b43449161285d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680490491199 2023-04-03 02:54:51,258 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:51,265 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:51,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:51,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:51,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:51,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:51,269 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:51,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:51,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:51,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:51,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:51,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:51,270 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,270 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:51,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:51,271 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,271 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:51,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:51,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:51,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:51,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:51,272 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:51,272 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:51,272 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:51,272 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:51,272 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:51,272 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:51,272 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:51,272 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 02:54:51,273 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:51,273 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:51,273 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:51,273 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:51,273 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:51,273 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:51,280 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/c618691566884fee96a3a0d2cf4c4811 2023-04-03 02:54:51,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:51,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:51,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490551341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490551372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490551377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490551377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490551378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490551478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490551481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490551483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490551483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490551483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,528 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/18842a11cbd6493fbd45879afcc889ba as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/18842a11cbd6493fbd45879afcc889ba 2023-04-03 02:54:51,534 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into 18842a11cbd6493fbd45879afcc889ba(size=11.8 K), total size for store is 32.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:51,534 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:51,534 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=13, startTime=1680490491035; duration=0sec 2023-04-03 02:54:51,534 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:54:51,534 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:51,535 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33520 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:51,535 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:54:51,535 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:51,535 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/494aca889df8480f9fed66526ff19cc4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c9845fd868944621960d01401c265dbe, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/434d0e6cb89f455d960dbbb28b0cda74] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=32.7 K 2023-04-03 02:54:51,536 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 494aca889df8480f9fed66526ff19cc4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680490490469 2023-04-03 02:54:51,537 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c9845fd868944621960d01401c265dbe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680490490581 2023-04-03 02:54:51,537 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 434d0e6cb89f455d960dbbb28b0cda74, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680490491199 2023-04-03 02:54:51,543 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#219 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:51,667 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/1a46f3358d514955ad19ebea007efba7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/1a46f3358d514955ad19ebea007efba7 2023-04-03 02:54:51,671 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into 1a46f3358d514955ad19ebea007efba7(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:51,672 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:51,672 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=13, startTime=1680490491250; duration=0sec 2023-04-03 02:54:51,672 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:51,672 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:51,673 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33520 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:51,673 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:54:51,673 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:51,673 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/18842a11cbd6493fbd45879afcc889ba, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/5a7fcbab4c434806990ce75ff7408ddd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8f90f1049d4c4d36a2404ea8e5bc9522] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=32.7 K 2023-04-03 02:54:51,673 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 18842a11cbd6493fbd45879afcc889ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680490490469 2023-04-03 02:54:51,674 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 5a7fcbab4c434806990ce75ff7408ddd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680490490581 2023-04-03 02:54:51,674 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 8f90f1049d4c4d36a2404ea8e5bc9522, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680490491199 2023-04-03 02:54:51,679 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#220 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:51,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490551682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490551683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490551687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,689 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/e3e352849b774c23af3952af4d75c86e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e3e352849b774c23af3952af4d75c86e 2023-04-03 02:54:51,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490551688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490551688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,693 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/0ec2502211434a7d85e26caa5ec3fd2d 2023-04-03 02:54:51,704 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into e3e352849b774c23af3952af4d75c86e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:51,704 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:51,704 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=13, startTime=1680490491250; duration=0sec 2023-04-03 02:54:51,704 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:51,960 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/d7a778598d0e4b1e96874ab4766ee681 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d7a778598d0e4b1e96874ab4766ee681 2023-04-03 02:54:51,965 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into d7a778598d0e4b1e96874ab4766ee681(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:51,966 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:51,966 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=13, startTime=1680490491250; duration=0sec 2023-04-03 02:54:51,966 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:51,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490551986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490551986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490551989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490551990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:51,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:51,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490551994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,110 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/c452c5be024148728de001c9c6be0f12 2023-04-03 02:54:52,117 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/c618691566884fee96a3a0d2cf4c4811 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c618691566884fee96a3a0d2cf4c4811 2023-04-03 02:54:52,122 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c618691566884fee96a3a0d2cf4c4811, entries=150, sequenceid=150, filesize=11.8 K 2023-04-03 02:54:52,123 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/0ec2502211434a7d85e26caa5ec3fd2d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0ec2502211434a7d85e26caa5ec3fd2d 2023-04-03 02:54:52,129 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0ec2502211434a7d85e26caa5ec3fd2d, entries=150, sequenceid=150, filesize=11.8 K 2023-04-03 02:54:52,130 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/c452c5be024148728de001c9c6be0f12 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c452c5be024148728de001c9c6be0f12 2023-04-03 02:54:52,148 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c452c5be024148728de001c9c6be0f12, entries=150, sequenceid=150, filesize=11.8 K 2023-04-03 02:54:52,149 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=60.38 KB/61830 for ca59c59c9ebb324c8ea1a6deda6d6369 in 877ms, sequenceid=150, compaction requested=false 2023-04-03 02:54:52,149 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:52,149 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:52,149 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:52,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:52,153 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:52,153 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:52,154 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:52,154 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4239fceb[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:52,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:52,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:52,158 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:52,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:52,158 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:52,158 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:52,158 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:52,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:52,160 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:52,160 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:52,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,160 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:52,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:52,163 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:52,163 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:52,163 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:52,163 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:52,163 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:52,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:52,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:52,165 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:52,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:52,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:52,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:52,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:52,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:52,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:52,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:52,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:52,170 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:52,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:52,170 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:52,170 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:52,170 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:52,170 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:52,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:52,171 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:52,171 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:52,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:52,271 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:52,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:52,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:52,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:52,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:52,274 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:52,274 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:52,274 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:52,274 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:52,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:52,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:52,277 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,277 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:52,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:52,277 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,277 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:52,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:52,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:52,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:52,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:52,278 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:52,278 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:52,278 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:52,279 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:52,279 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:52,279 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:52,279 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:52,279 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:54:52,280 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:52,280 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:52,280 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:52,280 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:52,280 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:52,280 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:52,411 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 02:54:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:52,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:52,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490552503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490552527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490552527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490552527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490552528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490552630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490552632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490552632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490552632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490552632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,694 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/c691d2f954834682b9bca5320a181816 2023-04-03 02:54:52,706 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/7c5cb48d542f4cb3ab13c046c51b42dc 2023-04-03 02:54:52,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490552833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490552834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490552835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490552836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:52,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:52,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490552836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,129 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/b29c3f5baa384856996d692edfd00e4a 2023-04-03 02:54:53,133 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/c691d2f954834682b9bca5320a181816 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c691d2f954834682b9bca5320a181816 2023-04-03 02:54:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490553138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490553138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490553138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490553139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,141 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c691d2f954834682b9bca5320a181816, entries=150, sequenceid=166, filesize=11.8 K 2023-04-03 02:54:53,141 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/7c5cb48d542f4cb3ab13c046c51b42dc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7c5cb48d542f4cb3ab13c046c51b42dc 2023-04-03 02:54:53,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490553141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,145 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7c5cb48d542f4cb3ab13c046c51b42dc, entries=150, sequenceid=166, filesize=11.8 K 2023-04-03 02:54:53,146 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/b29c3f5baa384856996d692edfd00e4a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/b29c3f5baa384856996d692edfd00e4a 2023-04-03 02:54:53,149 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/b29c3f5baa384856996d692edfd00e4a, entries=150, sequenceid=166, filesize=11.8 K 2023-04-03 02:54:53,150 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ca59c59c9ebb324c8ea1a6deda6d6369 in 871ms, sequenceid=166, compaction requested=true 2023-04-03 02:54:53,150 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:53,150 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:53,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50411598[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:53,153 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:53,153 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,155 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:53,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:53,157 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:53,157 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:53,157 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,157 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:53,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:53,160 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:53,160 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:53,160 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:53,160 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:53,160 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:53,163 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,163 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,163 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:53,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,164 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,169 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:53,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:53,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:53,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:53,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:53,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:53,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:53,170 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:53,170 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:53,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:53,270 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:53,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:53,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:53,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:53,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:53,273 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:53,273 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:53,273 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:53,273 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:53,274 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,274 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:53,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:53,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:53,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:53,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:53,276 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:53,276 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:53,276 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:53,276 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:53,276 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:53,276 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:53,276 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:53,277 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 02:54:53,277 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:53,277 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:53,277 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:53,277 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:53,277 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:53,277 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:53,285 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/0dd724da41a14388811b91332dc88b95 2023-04-03 02:54:53,298 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/07569886019c4db187084410ade7e28e 2023-04-03 02:54:53,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:53,645 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:53,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490553651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490553654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490553655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490553655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490553655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,710 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f194788520544a24bd0af09f8965d0ae 2023-04-03 02:54:53,715 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/0dd724da41a14388811b91332dc88b95 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0dd724da41a14388811b91332dc88b95 2023-04-03 02:54:53,719 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0dd724da41a14388811b91332dc88b95, entries=150, sequenceid=186, filesize=11.8 K 2023-04-03 02:54:53,720 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/07569886019c4db187084410ade7e28e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/07569886019c4db187084410ade7e28e 2023-04-03 02:54:53,723 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/07569886019c4db187084410ade7e28e, entries=150, sequenceid=186, filesize=11.8 K 2023-04-03 02:54:53,724 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f194788520544a24bd0af09f8965d0ae as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f194788520544a24bd0af09f8965d0ae 2023-04-03 02:54:53,728 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f194788520544a24bd0af09f8965d0ae, entries=150, sequenceid=186, filesize=11.8 K 2023-04-03 02:54:53,728 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=73.80 KB/75570 for ca59c59c9ebb324c8ea1a6deda6d6369 in 451ms, sequenceid=186, compaction requested=true 2023-04-03 02:54:53,729 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:53,729 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:53,730 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,730 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,731 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:53,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c8488a7[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:53,732 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:53,732 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,736 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:53,737 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,737 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:53,738 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:53,741 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:53,741 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:53,741 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:53,741 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:53,741 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:53,743 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,743 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,743 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,743 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,745 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,747 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:53,747 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:53,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:53,747 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:53,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:53,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:53,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:53,747 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:53,747 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,747 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,747 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:53,748 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:53,748 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,748 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,748 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,748 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:53,759 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 02:54:53,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:53,760 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:53,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:53,760 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:53,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:53,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:53,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490553771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490553771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490553771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490553772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490553773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,848 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:53,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:53,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:53,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:53,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:53,852 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:53,852 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:53,852 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:53,852 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:53,852 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,852 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,855 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:53,855 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:53,855 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,855 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:53,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:53,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:53,857 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:53,857 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:53,857 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:53,857 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:53,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:53,858 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,858 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:53,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ab43494[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:53,860 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:53,860 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,861 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,861 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,861 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:53,861 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:53,861 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:53,862 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:53,863 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:53,863 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,863 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,863 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:53,866 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:53,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:53,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:53,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:53,866 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:53,868 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,868 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,868 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:53,869 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,872 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,872 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:53,872 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:53,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:53,872 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:53,872 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:53,872 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:53,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:53,873 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,873 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,873 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,873 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,873 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:53,873 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:53,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:53,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490553874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490553874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490553874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490553874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:53,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490553877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,973 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:53,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:53,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:53,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:53,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:53,977 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:53,977 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:53,977 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:53,977 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:53,977 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,977 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,978 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:53,978 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:53,979 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,979 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:53,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:53,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:53,980 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:53,980 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:53,980 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:53,980 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:53,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27674b8d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:53,983 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:53,984 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,988 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:53,988 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,988 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:53,990 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:53,992 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:53,992 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:53,992 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:53,992 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:53,992 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:53,994 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,994 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,994 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:53,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:53,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,995 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,997 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:53,997 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:53,997 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:53,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:53,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:53,998 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:53,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:53,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:53,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:53,998 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:53,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:53,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:53,998 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:53,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:54,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490554077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490554078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490554079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490554079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490554080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,098 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:54,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:54,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:54,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:54,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:54,101 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:54,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:54,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:54,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,104 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,104 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:54,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:54,104 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,104 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:54,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:54,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:54,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:54,105 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:54,105 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,105 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:54,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:54,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:54,106 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,106 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:54,106 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:54,106 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:54,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:54,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:54,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:54,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,107 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,108 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,108 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:54,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd2772c[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:54,109 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:54,109 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,110 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:54,110 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:54,111 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:54,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:54,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:54,129 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:54,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:54,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:54,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:54,129 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:54,131 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,131 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,131 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:54,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,133 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:54,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:54,137 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:54,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:54,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:54,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:54,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:54,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,137 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:54,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,137 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:54,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:54,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/0e63d68321414df18a947c75a9b1735f 2023-04-03 02:54:54,237 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:54,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:54,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:54,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:54,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:54,241 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:54,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:54,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:54,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,242 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:54,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:54,243 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,243 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:54,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:54,244 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,244 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:54,244 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:54,244 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:54,247 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,247 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,247 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@274b09c1[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:54,249 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:54,249 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,250 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:54,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:54,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:54,250 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,250 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:54,252 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,252 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:54,252 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,252 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:54,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:54,255 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:54,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:54,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:54,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:54,255 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:54,256 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,256 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,256 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:54,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,258 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:54,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:54,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:54,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:54,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:54,261 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:54,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:54,261 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:54,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:54,261 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:54,362 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:54,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:54,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:54,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:54,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:54,367 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:54,367 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,367 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:54,367 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:54,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,369 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:54,369 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:54,369 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,369 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:54,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:54,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:54,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:54,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:54,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:54,372 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,372 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:54,372 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:54,372 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:54,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:54,373 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd08928[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:54,375 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:54,376 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,377 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:54,377 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,377 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:54,379 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,379 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:54,380 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:54,380 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490554380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490554382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:54,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:54,384 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:54,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:54,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:54,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:54,384 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:54,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490554385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:54,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490554386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,389 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490554388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:54,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:54,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:54,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,393 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:54,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:54,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:54,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:54,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,393 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:54,393 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:54,508 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:54,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:54,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:54,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:54,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:54,512 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:54,512 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,512 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:54,512 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:54,513 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,513 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,514 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,514 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:54,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:54,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:54,524 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,524 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:54,524 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:54,524 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,527 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f6841c[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:54,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:54,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:54,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:54,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:54,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:54,533 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:54,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:54,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:54,536 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:54,536 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:54,536 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:54,536 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:54,536 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:54,538 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,538 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,538 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:54,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,539 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,541 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,550 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,550 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,550 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,550 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:54,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:54,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:54,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:54,550 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:54,550 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,550 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:54,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:54,550 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:54,550 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,551 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,551 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:54,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:54,579 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/ab07476b983c46eb8fe02a696068c5bd 2023-04-03 02:54:54,589 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/dec7ca8cf4dd4cae9c44f0077290f0b3 2023-04-03 02:54:54,593 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/0e63d68321414df18a947c75a9b1735f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0e63d68321414df18a947c75a9b1735f 2023-04-03 02:54:54,597 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0e63d68321414df18a947c75a9b1735f, entries=250, sequenceid=202, filesize=16.4 K 2023-04-03 02:54:54,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/ab07476b983c46eb8fe02a696068c5bd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/ab07476b983c46eb8fe02a696068c5bd 2023-04-03 02:54:54,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/ab07476b983c46eb8fe02a696068c5bd, entries=150, sequenceid=202, filesize=11.8 K 2023-04-03 02:54:54,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/dec7ca8cf4dd4cae9c44f0077290f0b3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/dec7ca8cf4dd4cae9c44f0077290f0b3 2023-04-03 02:54:54,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/dec7ca8cf4dd4cae9c44f0077290f0b3, entries=150, sequenceid=202, filesize=11.8 K 2023-04-03 02:54:54,614 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for ca59c59c9ebb324c8ea1a6deda6d6369 in 855ms, sequenceid=202, compaction requested=true 2023-04-03 02:54:54,614 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:54,614 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:54,614 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:54:54,614 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:54:54,614 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:54,614 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:54,616 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 65118 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:54:54,616 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:54:54,616 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,616 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/1a46f3358d514955ad19ebea007efba7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c618691566884fee96a3a0d2cf4c4811, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c691d2f954834682b9bca5320a181816, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0dd724da41a14388811b91332dc88b95, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0e63d68321414df18a947c75a9b1735f] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=63.6 K 2023-04-03 02:54:54,617 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60338 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:54:54,617 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:54:54,617 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,617 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d7a778598d0e4b1e96874ab4766ee681, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0ec2502211434a7d85e26caa5ec3fd2d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7c5cb48d542f4cb3ab13c046c51b42dc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/07569886019c4db187084410ade7e28e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/ab07476b983c46eb8fe02a696068c5bd] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=58.9 K 2023-04-03 02:54:54,617 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 1a46f3358d514955ad19ebea007efba7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680490490581 2023-04-03 02:54:54,618 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d7a778598d0e4b1e96874ab4766ee681, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680490490581 2023-04-03 02:54:54,618 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c618691566884fee96a3a0d2cf4c4811, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680490491225 2023-04-03 02:54:54,618 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 0ec2502211434a7d85e26caa5ec3fd2d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680490491225 2023-04-03 02:54:54,619 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7c5cb48d542f4cb3ab13c046c51b42dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680490491343 2023-04-03 02:54:54,619 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c691d2f954834682b9bca5320a181816, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680490491343 2023-04-03 02:54:54,620 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 0dd724da41a14388811b91332dc88b95, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680490492501 2023-04-03 02:54:54,620 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 07569886019c4db187084410ade7e28e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680490492501 2023-04-03 02:54:54,620 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 0e63d68321414df18a947c75a9b1735f, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680490493648 2023-04-03 02:54:54,622 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ab07476b983c46eb8fe02a696068c5bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680490493648 2023-04-03 02:54:54,650 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#231 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:54,651 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:54,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:54,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:54,664 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#232 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:54,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:54,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:54,682 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:54,682 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,682 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:54,682 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:54,682 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,683 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,684 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:54,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:54,684 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:54,684 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,684 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:54,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:54,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:54,708 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:54,708 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,708 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:54,742 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/4a853b1b4d134d6d8af38c18092cba82 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/4a853b1b4d134d6d8af38c18092cba82 2023-04-03 02:54:54,744 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:54,744 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:54,744 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,744 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:54,744 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 02:54:54,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:54,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:54,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:54,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:54,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:54,745 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:54,752 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into 4a853b1b4d134d6d8af38c18092cba82(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:54,752 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:54,752 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=11, startTime=1680490494614; duration=0sec 2023-04-03 02:54:54,752 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:54,752 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:54:54,757 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60338 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:54:54,757 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:54:54,757 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,757 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e3e352849b774c23af3952af4d75c86e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c452c5be024148728de001c9c6be0f12, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/b29c3f5baa384856996d692edfd00e4a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f194788520544a24bd0af09f8965d0ae, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/dec7ca8cf4dd4cae9c44f0077290f0b3] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=58.9 K 2023-04-03 02:54:54,758 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e3e352849b774c23af3952af4d75c86e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680490490581 2023-04-03 02:54:54,758 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c452c5be024148728de001c9c6be0f12, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680490491225 2023-04-03 02:54:54,759 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting b29c3f5baa384856996d692edfd00e4a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680490491343 2023-04-03 02:54:54,759 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting f194788520544a24bd0af09f8965d0ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680490492501 2023-04-03 02:54:54,759 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting dec7ca8cf4dd4cae9c44f0077290f0b3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680490493648 2023-04-03 02:54:54,765 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/a75cf1f3c27d467083ec43cb07365ea6 2023-04-03 02:54:54,786 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#235 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:54,808 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/bda2cd5ace9a4336a2b743214a5a9545 2023-04-03 02:54:54,818 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/a987b5bd18ff42448ab3297a58a77757 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/a987b5bd18ff42448ab3297a58a77757 2023-04-03 02:54:54,826 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into a987b5bd18ff42448ab3297a58a77757(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:54,827 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:54,827 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=11, startTime=1680490494614; duration=0sec 2023-04-03 02:54:54,827 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:54,838 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/8fd622ecd11841a08758393b1a3144d7 2023-04-03 02:54:54,843 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/a75cf1f3c27d467083ec43cb07365ea6 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/a75cf1f3c27d467083ec43cb07365ea6 2023-04-03 02:54:54,850 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/a75cf1f3c27d467083ec43cb07365ea6, entries=150, sequenceid=218, filesize=11.8 K 2023-04-03 02:54:54,850 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/bda2cd5ace9a4336a2b743214a5a9545 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/bda2cd5ace9a4336a2b743214a5a9545 2023-04-03 02:54:54,856 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/bda2cd5ace9a4336a2b743214a5a9545, entries=150, sequenceid=218, filesize=11.8 K 2023-04-03 02:54:54,857 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/8fd622ecd11841a08758393b1a3144d7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8fd622ecd11841a08758393b1a3144d7 2023-04-03 02:54:54,863 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8fd622ecd11841a08758393b1a3144d7, entries=150, sequenceid=218, filesize=11.8 K 2023-04-03 02:54:54,864 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for ca59c59c9ebb324c8ea1a6deda6d6369 in 120ms, sequenceid=218, compaction requested=false 2023-04-03 02:54:54,864 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:54,864 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,864 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:54,864 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:54,864 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,864 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:54,864 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:54,866 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,866 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,866 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:54,868 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:54,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d71ee19[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:54,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:54,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,872 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:54,872 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:54,873 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:54,873 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,873 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:54,874 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:54,878 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:54,878 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:54,878 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:54,878 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:54,878 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:54,879 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,879 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,879 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,880 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,882 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:54,885 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,885 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:54,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:54,885 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,885 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:54,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:54,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:54,885 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,885 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,886 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,886 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,886 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,886 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:54,886 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:54,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:54,895 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:54:54,895 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:54,895 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:54,895 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:54,895 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:54,895 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:54,895 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:54,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490494906, max timestamp=1680490494914Num uniques 859; 2023-04-03 02:54:54,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:54:54,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490494906, max timestamp=1680490494914Num uniques 859; 2023-04-03 02:54:54,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490494906, max timestamp=1680490494914Num uniques 859; 2023-04-03 02:54:54,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:54:54,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:54:54,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490554929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490554930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490554931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490554933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:54,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490554934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,986 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:54,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:54,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:54,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:54,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:54,990 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:54,990 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,990 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:54,990 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:54,991 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,991 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,993 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,993 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:54,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:54,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:54,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:54,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:54,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:54,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:54,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:54,996 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,996 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:54,996 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:54,996 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:54,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220be177[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:54,999 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:55,000 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,008 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:55,008 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,008 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:55,009 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,009 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:55,010 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,010 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:55,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:55,013 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:55,013 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:55,013 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:55,013 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:55,013 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:55,018 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,018 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,018 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,019 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,020 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:55,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:55,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:55,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,023 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,024 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:55,024 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:55,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490555035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490555036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490555036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490555040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490555040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,124 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:55,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:55,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:55,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:55,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:55,127 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:55,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:55,128 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:55,128 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,128 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,130 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,130 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:55,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:55,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:55,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:55,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:55,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:55,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:55,132 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,132 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:55,132 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:55,132 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:55,134 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,135 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/61efeaaca5e249ac851c3c8a6f53ecd5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/61efeaaca5e249ac851c3c8a6f53ecd5 2023-04-03 02:54:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:55,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@abab5d7[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:55,136 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:55,136 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,138 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:55,138 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:55,139 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into 61efeaaca5e249ac851c3c8a6f53ecd5(size=12.2 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:55,139 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:55,139 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=11, startTime=1680490494614; duration=0sec 2023-04-03 02:54:55,139 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,140 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:55,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:55,143 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:55,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:55,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:55,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:55,143 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:55,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,145 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,146 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,151 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,151 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,151 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:55,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:55,151 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,152 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:55,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,152 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:55,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490555239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490555239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490555240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490555244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490555245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,252 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:55,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:55,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:55,255 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:55,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:55,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:55,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:55,258 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:55,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:55,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:55,259 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,259 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:55,260 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:55,260 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:55,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:55,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f6906f8[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:55,263 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:55,263 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,265 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:55,266 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,266 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:55,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,267 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:55,267 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,267 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:55,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:55,269 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:55,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:55,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:55,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:55,269 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:55,271 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,271 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,271 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,272 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,273 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:55,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:55,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,275 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:55,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,276 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:55,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:55,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/8b537e002f2d4e9aa5f61253ea721ad1 2023-04-03 02:54:55,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/3629a3bdf3bd4e8882b613c95b62d98e 2023-04-03 02:54:55,376 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:55,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:55,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:55,379 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:55,379 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,379 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:55,380 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:55,380 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,380 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,382 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,382 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,382 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:55,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:55,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:55,384 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,384 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:55,384 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:55,384 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:55,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:55,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:55,388 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:55,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dba202a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:55,388 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,389 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:55,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:55,389 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,389 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:55,390 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,390 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:55,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:55,393 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:55,393 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:55,393 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:55,393 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:55,393 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:55,395 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,395 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,395 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,395 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,398 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,398 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,398 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,399 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,399 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,399 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,399 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:55,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:55,399 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,399 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:55,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,399 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,399 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,399 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:55,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,499 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:55,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:55,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:55,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:55,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:55,502 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:55,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:55,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:55,503 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,503 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,504 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,504 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,504 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:55,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:55,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:55,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:55,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:55,506 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,506 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:55,506 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:55,506 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:55,508 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,508 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:55,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20c762b3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:55,510 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:55,511 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,518 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:55,518 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:55,518 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:55,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:55,519 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,519 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:55,519 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:55,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:55,522 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:55,522 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:55,522 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:55,522 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:55,522 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:55,524 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,524 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,524 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,524 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,526 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:55,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:55,528 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,528 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:55,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,528 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:55,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490555541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490555542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490555542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490555547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:55,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490555548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,628 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:55,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:55,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:55,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:55,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:55,632 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:55,632 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,632 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:55,632 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:55,633 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,633 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,637 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,637 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,638 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,638 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:55,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:55,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:55,639 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:55,639 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,640 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:55,640 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:55,640 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:55,640 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,640 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:55,640 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:55,640 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,640 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:55,640 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:55,640 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,640 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:55,640 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:55,644 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,644 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,645 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,645 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:55,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f0c6f71[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:55,652 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:55,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,654 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:55,654 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,655 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,656 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:55,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:55,660 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:55,660 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:55,660 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:55,660 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:55,660 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:55,665 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:55,665 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,665 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:55,666 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:55,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:55,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:55,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:55,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,671 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:55,671 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:55,671 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:55,671 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:55,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:55,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/9ee2ac2d488f466689a6c9da31e91aa9 2023-04-03 02:54:55,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/8b537e002f2d4e9aa5f61253ea721ad1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8b537e002f2d4e9aa5f61253ea721ad1 2023-04-03 02:54:55,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8b537e002f2d4e9aa5f61253ea721ad1, entries=250, sequenceid=231, filesize=16.4 K 2023-04-03 02:54:55,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/3629a3bdf3bd4e8882b613c95b62d98e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/3629a3bdf3bd4e8882b613c95b62d98e 2023-04-03 02:54:55,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/3629a3bdf3bd4e8882b613c95b62d98e, entries=150, sequenceid=231, filesize=11.8 K 2023-04-03 02:54:55,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/9ee2ac2d488f466689a6c9da31e91aa9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ee2ac2d488f466689a6c9da31e91aa9 2023-04-03 02:54:55,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ee2ac2d488f466689a6c9da31e91aa9, entries=150, sequenceid=231, filesize=11.8 K 2023-04-03 02:54:55,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ca59c59c9ebb324c8ea1a6deda6d6369 in 863ms, sequenceid=231, compaction requested=true 2023-04-03 02:54:55,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:55,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:55,758 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:55,758 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:55,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:55,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:55,759 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:55,759 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:54:55,759 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:55,759 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,759 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:54:55,759 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/4a853b1b4d134d6d8af38c18092cba82, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/a75cf1f3c27d467083ec43cb07365ea6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8b537e002f2d4e9aa5f61253ea721ad1] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=40.4 K 2023-04-03 02:54:55,759 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,759 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/61efeaaca5e249ac851c3c8a6f53ecd5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/bda2cd5ace9a4336a2b743214a5a9545, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/3629a3bdf3bd4e8882b613c95b62d98e] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=35.7 K 2023-04-03 02:54:55,760 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 4a853b1b4d134d6d8af38c18092cba82, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680490493648 2023-04-03 02:54:55,760 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 61efeaaca5e249ac851c3c8a6f53ecd5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680490493648 2023-04-03 02:54:55,760 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a75cf1f3c27d467083ec43cb07365ea6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680490493769 2023-04-03 02:54:55,760 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting bda2cd5ace9a4336a2b743214a5a9545, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680490493769 2023-04-03 02:54:55,760 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 8b537e002f2d4e9aa5f61253ea721ad1, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680490494889 2023-04-03 02:54:55,760 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 3629a3bdf3bd4e8882b613c95b62d98e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680490494893 2023-04-03 02:54:55,767 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#240 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:55,767 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#241 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:55,771 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:55,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:55,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:55,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:55,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:55,775 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:55,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:55,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:55,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,777 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:55,777 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,777 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:55,777 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:55,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:55,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:55,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:55,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:55,780 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:55,780 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-03 02:54:55,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:55,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:55,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:55,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:55,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:55,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:55,790 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/cff0733d39844d33a5e17a0e7ec3adbd 2023-04-03 02:54:56,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:56,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:56,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490556051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490556055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490556055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490556057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490556056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490556157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490556162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490556162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490556162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490556163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,179 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/13eccae1feac4c49b75818433f4c1be9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/13eccae1feac4c49b75818433f4c1be9 2023-04-03 02:54:56,185 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/faee7a1237194568aa63bd94ea82b3bf as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/faee7a1237194568aa63bd94ea82b3bf 2023-04-03 02:54:56,187 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into 13eccae1feac4c49b75818433f4c1be9(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:56,187 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:56,187 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=13, startTime=1680490495758; duration=0sec 2023-04-03 02:54:56,187 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:56,187 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:56,188 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:56,189 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:54:56,189 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,189 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/a987b5bd18ff42448ab3297a58a77757, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8fd622ecd11841a08758393b1a3144d7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ee2ac2d488f466689a6c9da31e91aa9] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=35.7 K 2023-04-03 02:54:56,189 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a987b5bd18ff42448ab3297a58a77757, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680490493648 2023-04-03 02:54:56,190 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 8fd622ecd11841a08758393b1a3144d7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680490493769 2023-04-03 02:54:56,190 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 9ee2ac2d488f466689a6c9da31e91aa9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680490494893 2023-04-03 02:54:56,191 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into faee7a1237194568aa63bd94ea82b3bf(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:56,191 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:56,191 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=13, startTime=1680490495758; duration=0sec 2023-04-03 02:54:56,191 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:56,196 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:56,202 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/5bf493a7c1a34d86a1722cc7d4c1209d 2023-04-03 02:54:56,206 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/7c8012f1fd304bf999e840cfa781543d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/7c8012f1fd304bf999e840cfa781543d 2023-04-03 02:54:56,217 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into 7c8012f1fd304bf999e840cfa781543d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:56,217 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:56,217 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=13, startTime=1680490495758; duration=0sec 2023-04-03 02:54:56,217 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:56,218 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/9a2f6c414f514606a0a7eaa217cca75e 2023-04-03 02:54:56,223 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/cff0733d39844d33a5e17a0e7ec3adbd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cff0733d39844d33a5e17a0e7ec3adbd 2023-04-03 02:54:56,228 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cff0733d39844d33a5e17a0e7ec3adbd, entries=150, sequenceid=266, filesize=11.9 K 2023-04-03 02:54:56,229 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/5bf493a7c1a34d86a1722cc7d4c1209d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/5bf493a7c1a34d86a1722cc7d4c1209d 2023-04-03 02:54:56,233 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/5bf493a7c1a34d86a1722cc7d4c1209d, entries=150, sequenceid=266, filesize=11.9 K 2023-04-03 02:54:56,234 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/9a2f6c414f514606a0a7eaa217cca75e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9a2f6c414f514606a0a7eaa217cca75e 2023-04-03 02:54:56,239 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9a2f6c414f514606a0a7eaa217cca75e, entries=150, sequenceid=266, filesize=11.9 K 2023-04-03 02:54:56,240 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for ca59c59c9ebb324c8ea1a6deda6d6369 in 460ms, sequenceid=266, compaction requested=false 2023-04-03 02:54:56,240 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:56,240 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:56,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:56,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:56,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:56,241 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:56,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3815f8ae[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:56,245 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:56,245 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,246 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:56,246 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,246 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:56,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:56,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:56,247 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:56,248 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:56,251 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:56,251 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:56,251 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:56,251 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:56,251 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:56,254 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,254 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,254 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:56,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:56,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:56,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:56,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:56,259 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,259 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:56,259 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:56,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,359 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:56,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:56,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:56,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:56,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 02:54:56,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:56,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:56,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:56,363 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:56,363 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:56,363 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:56,363 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:56,363 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:56,363 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:56,363 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:56,363 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:56,363 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:56,363 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,363 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,365 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:56,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:56,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:56,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:56,367 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:56,368 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,368 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:56,368 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:56,368 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,370 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,372 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/9d7365adccca4b7aab32121457545bbc 2023-04-03 02:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34a234f6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:56,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:56,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,375 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,375 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:56,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:56,376 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:56,376 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:56,377 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:56,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:56,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:56,380 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:56,380 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:56,380 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:56,380 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:56,380 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:56,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490556380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680490496377, max timestamp=1680490496384Num uniques 859; 2023-04-03 02:54:56,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:54:56,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490496377, max timestamp=1680490496384Num uniques 859; 2023-04-03 02:54:56,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:54:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490496377, max timestamp=1680490496384Num uniques 859; 2023-04-03 02:54:56,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:54:56,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:56,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:56,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:56,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,388 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:56,388 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:56,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:56,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:56,388 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,388 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,388 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,388 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,388 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490556403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490556403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490556403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490556404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490556487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,488 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:56,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:56,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:56,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:56,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:56,491 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:56,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:56,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:56,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:56,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,493 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:56,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:56,494 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,494 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:56,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:56,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:56,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:56,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:56,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:56,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:56,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:56,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:56,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:56,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:56,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:56,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:56,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:56,497 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,497 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,498 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,498 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:56,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2feb1a2b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:56,499 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:56,499 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,500 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,500 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:56,500 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:56,500 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:56,500 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,500 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:56,503 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:56,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:56,505 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:56,506 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:56,506 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:56,506 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:56,506 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:56,507 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,507 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,507 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,508 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490556507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490556507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,510 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490556507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490556507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,513 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,513 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,513 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:56,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:56,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:56,513 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,513 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,513 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,513 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,513 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,513 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:56,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:56,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:56,513 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:56,513 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:56,513 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,613 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:56,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:56,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:56,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:56,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:56,616 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:56,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:56,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:56,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:56,617 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,617 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,618 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,618 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:56,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:56,618 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,618 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:56,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:56,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:56,619 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:56,619 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:56,619 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:56,620 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,620 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:56,620 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:56,620 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:56,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,623 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33809f40[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:56,625 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:56,625 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,626 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,626 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:56,626 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:56,626 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:56,627 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,627 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:56,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:56,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:56,630 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:56,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:56,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:56,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:56,630 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:56,632 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,632 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,632 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,633 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,634 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,639 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:56,639 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,639 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,639 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,639 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,639 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,639 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:56,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:56,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:56,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:56,640 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:56,639 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,639 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,640 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,640 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:56,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490556690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490556712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490556712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490556713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490556713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,740 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:56,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:56,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:56,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:56,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:56,743 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:56,743 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:56,743 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:56,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:56,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,745 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:56,745 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:56,745 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:56,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:56,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:56,747 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,747 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:56,747 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:56,747 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:56,749 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,749 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:56,751 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:56,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24c7d116[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:56,751 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,752 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,752 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:56,752 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:56,752 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:56,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:56,754 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,754 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:56,754 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,754 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:56,758 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:56,758 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:56,758 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:56,758 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:56,758 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:56,760 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,760 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,760 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,761 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,763 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:56,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,766 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:56,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:56,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:56,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,766 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:56,766 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:56,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,783 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/73ee2df950c64b9c910d99a5925cd877 2023-04-03 02:54:56,866 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:56,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:56,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:56,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:56,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:56,874 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:56,875 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:56,875 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:56,875 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:56,875 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,875 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,876 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,876 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:56,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:56,877 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,877 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:56,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:56,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:56,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:56,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:56,879 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,879 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:56,879 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:56,879 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:56,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:56,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:56,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:56,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:56,882 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,882 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,883 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,883 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:56,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e754fc[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:56,888 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:56,888 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,889 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,890 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:56,890 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,890 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:56,890 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:56,890 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:56,894 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:56,894 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,894 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:56,894 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:56,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:56,896 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:56,896 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:56,897 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:56,897 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:56,897 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:56,898 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:56,898 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,898 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:56,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,899 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,900 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,902 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,902 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,902 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:56,902 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:56,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:56,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:56,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:56,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:56,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:56,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:56,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:56,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:56,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:56,904 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:56,904 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:56,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:56,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:56,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490556992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,004 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:57,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:57,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:57,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:57,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:57,007 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:57,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:57,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:57,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:57,008 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,008 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,010 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:57,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:57,010 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:57,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:57,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:57,012 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:57,012 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:57,012 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:57,012 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:57,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,014 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:57,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:57,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:57,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@762a2ca9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:57,017 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,017 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:57,017 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:57,017 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:57,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,017 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490557015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,017 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:57,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490557015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490557016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490557017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:57,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:57,019 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:57,019 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:57,019 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:57,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:57,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:57,021 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:57,021 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:57,021 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:57,021 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:57,021 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:57,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:57,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,025 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:57,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:57,026 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,028 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,028 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,028 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,028 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,028 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:57,028 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:57,028 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:57,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:57,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:57,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:57,028 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,028 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:57,028 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,029 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:57,029 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:57,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:57,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:57,129 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:57,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:57,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:57,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:57,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:57,132 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:57,132 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:57,132 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:57,132 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:57,132 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,132 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,134 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:57,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:57,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:57,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:57,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:57,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:57,136 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:57,136 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:57,136 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:57,136 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:57,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:57,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:57,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:57,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:57,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41e7d7b3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:57,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:57,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,141 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,142 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:57,142 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:57,142 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:57,143 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:57,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,143 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:57,143 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:57,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:57,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:57,143 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:57,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:57,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:57,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:57,144 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:57,145 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,145 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:57,145 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:57,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:57,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:57,146 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,147 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,150 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:57,150 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:57,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:57,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:57,150 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,150 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:57,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:57,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:57,150 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,150 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:57,150 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:57,150 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:57,150 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,150 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,150 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:57,150 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:57,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/4d5665842112474cb7f54d7579aaa408 2023-04-03 02:54:57,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/9d7365adccca4b7aab32121457545bbc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9d7365adccca4b7aab32121457545bbc 2023-04-03 02:54:57,202 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9d7365adccca4b7aab32121457545bbc, entries=200, sequenceid=282, filesize=14.3 K 2023-04-03 02:54:57,203 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/73ee2df950c64b9c910d99a5925cd877 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/73ee2df950c64b9c910d99a5925cd877 2023-04-03 02:54:57,207 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/73ee2df950c64b9c910d99a5925cd877, entries=150, sequenceid=282, filesize=11.9 K 2023-04-03 02:54:57,208 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/4d5665842112474cb7f54d7579aaa408 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/4d5665842112474cb7f54d7579aaa408 2023-04-03 02:54:57,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/4d5665842112474cb7f54d7579aaa408, entries=150, sequenceid=282, filesize=11.9 K 2023-04-03 02:54:57,213 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=116.29 KB/119080 for ca59c59c9ebb324c8ea1a6deda6d6369 in 851ms, sequenceid=282, compaction requested=true 2023-04-03 02:54:57,213 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:57,213 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:57,213 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:57,213 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:57,213 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:57,213 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:57,216 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:57,216 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39428 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:57,216 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:54:57,216 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:54:57,216 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:57,216 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:57,216 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/faee7a1237194568aa63bd94ea82b3bf, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/5bf493a7c1a34d86a1722cc7d4c1209d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/73ee2df950c64b9c910d99a5925cd877] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=36.1 K 2023-04-03 02:54:57,216 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/13eccae1feac4c49b75818433f4c1be9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cff0733d39844d33a5e17a0e7ec3adbd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9d7365adccca4b7aab32121457545bbc] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=38.5 K 2023-04-03 02:54:57,217 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting faee7a1237194568aa63bd94ea82b3bf, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680490494893 2023-04-03 02:54:57,217 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 13eccae1feac4c49b75818433f4c1be9, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680490494893 2023-04-03 02:54:57,217 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 5bf493a7c1a34d86a1722cc7d4c1209d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680490494926 2023-04-03 02:54:57,217 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting cff0733d39844d33a5e17a0e7ec3adbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680490494926 2023-04-03 02:54:57,217 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 73ee2df950c64b9c910d99a5925cd877, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680490496049 2023-04-03 02:54:57,218 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 9d7365adccca4b7aab32121457545bbc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680490496049 2023-04-03 02:54:57,223 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#249 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:57,225 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#250 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:57,245 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/c714ec82d888461d8d7d28bcf4d59156 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c714ec82d888461d8d7d28bcf4d59156 2023-04-03 02:54:57,250 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into c714ec82d888461d8d7d28bcf4d59156(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:57,250 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:57,250 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=13, startTime=1680490497213; duration=0sec 2023-04-03 02:54:57,250 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:57,250 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:54:57,250 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:57,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:57,251 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:54:57,252 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:54:57,252 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:57,252 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/7c8012f1fd304bf999e840cfa781543d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9a2f6c414f514606a0a7eaa217cca75e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/4d5665842112474cb7f54d7579aaa408] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=36.1 K 2023-04-03 02:54:57,252 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 7c8012f1fd304bf999e840cfa781543d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680490494893 2023-04-03 02:54:57,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:57,253 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 9a2f6c414f514606a0a7eaa217cca75e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680490494926 2023-04-03 02:54:57,253 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 4d5665842112474cb7f54d7579aaa408, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680490496049 2023-04-03 02:54:57,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:57,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:57,254 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:57,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:57,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:57,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:57,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,256 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:57,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:57,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:57,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:57,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:57,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:57,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:57,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:57,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:57,259 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#251 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:57,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:57,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:57,260 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:57,260 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:57,260 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=116.29 KB heapSize=325.61 KB 2023-04-03 02:54:57,260 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:57,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:57,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:57,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:57,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:57,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:57,278 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/15e9a48e5f3a49639bc26be87772b564 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/15e9a48e5f3a49639bc26be87772b564 2023-04-03 02:54:57,282 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/2f941fffe7fb46188dac6c2bc393255e 2023-04-03 02:54:57,288 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into 15e9a48e5f3a49639bc26be87772b564(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:57,288 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:57,288 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=13, startTime=1680490497213; duration=0sec 2023-04-03 02:54:57,288 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:57,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:57,498 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:57,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33720 deadline: 1680490557519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490557520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33692 deadline: 1680490557521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33756 deadline: 1680490557522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490557523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490557623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490557624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,653 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/142a7e82795244a7a6976ccd4482c505 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/142a7e82795244a7a6976ccd4482c505 2023-04-03 02:54:57,659 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into 142a7e82795244a7a6976ccd4482c505(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:57,659 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:57,659 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=13, startTime=1680490497213; duration=0sec 2023-04-03 02:54:57,659 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:57,706 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/7e70cff08f754ac2a247a78d87d7d199 2023-04-03 02:54:57,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490557825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:57,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:57,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490557828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,120 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/360dc7073b7e4a5381f6124aab180189 2023-04-03 02:54:58,124 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/2f941fffe7fb46188dac6c2bc393255e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2f941fffe7fb46188dac6c2bc393255e 2023-04-03 02:54:58,127 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2f941fffe7fb46188dac6c2bc393255e, entries=150, sequenceid=316, filesize=11.9 K 2023-04-03 02:54:58,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/7e70cff08f754ac2a247a78d87d7d199 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7e70cff08f754ac2a247a78d87d7d199 2023-04-03 02:54:58,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490558127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,132 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7e70cff08f754ac2a247a78d87d7d199, entries=150, sequenceid=316, filesize=11.9 K 2023-04-03 02:54:58,133 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/360dc7073b7e4a5381f6124aab180189 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/360dc7073b7e4a5381f6124aab180189 2023-04-03 02:54:58,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490558132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,137 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/360dc7073b7e4a5381f6124aab180189, entries=150, sequenceid=316, filesize=11.9 K 2023-04-03 02:54:58,140 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~116.29 KB/119080, heapSize ~325.56 KB/333376, currentSize=67.09 KB/68700 for ca59c59c9ebb324c8ea1a6deda6d6369 in 880ms, sequenceid=316, compaction requested=false 2023-04-03 02:54:58,140 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:58,140 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:58,140 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,141 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:58,141 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:58,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,142 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:58,142 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:58,142 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:58,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:58,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bb2f1f8[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:58,144 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:58,144 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:58,145 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:58,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:58,146 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,146 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:58,147 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:58,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:58,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:58,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:58,150 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:58,150 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:58,150 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:58,150 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:58,150 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:58,153 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:58,153 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:58,153 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:58,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:58,153 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,155 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:58,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:58,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:58,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:58,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:58,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:58,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:58,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:58,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:58,160 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:58,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:58,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:58,160 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:58,160 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:58,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:58,261 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:58,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:58,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:58,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:58,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:58,264 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:58,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:58,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:58,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:58,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:58,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:58,265 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,265 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:58,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:58,266 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,266 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:58,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:58,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:58,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:58,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:58,267 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:58,267 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:58,267 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:58,267 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:58,267 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:58,267 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:58,267 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:58,267 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 02:54:58,268 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:58,268 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:58,268 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:58,268 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:58,268 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:58,268 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:58,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:58,527 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:58,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490558544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33770 deadline: 1680490558544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490558546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490558635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490558636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490558647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490558647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490558650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,676 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/088aa43e292a4460834a3dc5504df986 2023-04-03 02:54:58,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490558850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490558850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:33750 deadline: 1680490558852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,089 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/76436b28b8e54fff88f9ef510cb78a10 2023-04-03 02:54:59,101 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/ddbf48134e7e4cbfb1d99bd9629bae73 2023-04-03 02:54:59,105 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/088aa43e292a4460834a3dc5504df986 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/088aa43e292a4460834a3dc5504df986 2023-04-03 02:54:59,109 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/088aa43e292a4460834a3dc5504df986, entries=150, sequenceid=331, filesize=11.9 K 2023-04-03 02:54:59,110 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/76436b28b8e54fff88f9ef510cb78a10 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/76436b28b8e54fff88f9ef510cb78a10 2023-04-03 02:54:59,113 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/76436b28b8e54fff88f9ef510cb78a10, entries=150, sequenceid=331, filesize=11.9 K 2023-04-03 02:54:59,114 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/ddbf48134e7e4cbfb1d99bd9629bae73 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/ddbf48134e7e4cbfb1d99bd9629bae73 2023-04-03 02:54:59,117 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/ddbf48134e7e4cbfb1d99bd9629bae73, entries=150, sequenceid=331, filesize=11.9 K 2023-04-03 02:54:59,118 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ca59c59c9ebb324c8ea1a6deda6d6369 in 851ms, sequenceid=331, compaction requested=true 2023-04-03 02:54:59,118 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:59,118 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:59,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60c90ec6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:59,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:59,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,123 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:59,123 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:59,123 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:59,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:59,125 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,125 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:59,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:59,129 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:59,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:59,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:59,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:59,129 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:59,130 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,130 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,130 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,131 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,132 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,135 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:59,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:59,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,135 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:59,135 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:59,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:59,155 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-03 02:54:59,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:59,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:59,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:59,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:59,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:59,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:54:59,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490559170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490559171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490559171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/6d6aec2731474295828d31c5a683d1b0 2023-04-03 02:54:59,235 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:59,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:59,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:59,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:59,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:59,239 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:59,239 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,239 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:59,239 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:59,239 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,240 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,241 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:59,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:59,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:59,243 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,243 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:59,243 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:59,243 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:59,245 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,245 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,246 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:59,248 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:59,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47212912[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:59,248 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,249 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:59,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:59,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:59,249 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,249 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:59,250 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:59,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:59,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:59,253 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:59,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:59,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:59,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:59,253 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:59,255 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,255 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,255 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,258 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,266 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:59,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:59,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,267 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:59,267 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:59,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:59,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490559275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490559275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490559276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,367 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:59,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:59,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:59,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:59,380 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:59,380 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,380 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:59,380 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:59,381 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,381 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,382 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:59,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:59,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:59,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:59,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:59,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:59,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:59,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:59,396 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:59,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:59,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:59,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:59,398 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,400 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3804240a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:59,401 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:59,402 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,403 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,403 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:59,403 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:59,403 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:59,403 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,404 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:59,406 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,406 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:59,406 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:59,411 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:59,412 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:59,412 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:59,412 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:59,412 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:59,413 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,413 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,413 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,420 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,422 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,426 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,426 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,426 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,426 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:59,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:59,426 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,427 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,427 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,427 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,427 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,427 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:59,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,427 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:59,427 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490559478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490559479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490559479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,527 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:59,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:59,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:59,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:59,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:59,530 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:59,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:59,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:59,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,532 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,532 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,533 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,533 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:59,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:59,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:59,534 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,534 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:59,534 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:59,534 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,535 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,536 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,536 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:59,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f53fc3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:59,537 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:59,537 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,538 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:59,539 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,539 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:59,540 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:59,543 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:59,543 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:59,543 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:59,543 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:59,543 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:59,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,546 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,547 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,549 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:59,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:59,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:59,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,549 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:59,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,549 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:59,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/d5a384bed07f492abe1d99b1925aaca4 2023-04-03 02:54:59,604 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f5a640372b174939a327a4bde6fe09ac 2023-04-03 02:54:59,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/6d6aec2731474295828d31c5a683d1b0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6d6aec2731474295828d31c5a683d1b0 2023-04-03 02:54:59,615 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6d6aec2731474295828d31c5a683d1b0, entries=150, sequenceid=352, filesize=11.9 K 2023-04-03 02:54:59,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/d5a384bed07f492abe1d99b1925aaca4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d5a384bed07f492abe1d99b1925aaca4 2023-04-03 02:54:59,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d5a384bed07f492abe1d99b1925aaca4, entries=150, sequenceid=352, filesize=11.9 K 2023-04-03 02:54:59,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f5a640372b174939a327a4bde6fe09ac as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f5a640372b174939a327a4bde6fe09ac 2023-04-03 02:54:59,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f5a640372b174939a327a4bde6fe09ac, entries=150, sequenceid=352, filesize=11.9 K 2023-04-03 02:54:59,630 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for ca59c59c9ebb324c8ea1a6deda6d6369 in 475ms, sequenceid=352, compaction requested=true 2023-04-03 02:54:59,630 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:59,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:59,630 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:54:59,630 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:54:59,631 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:59,631 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:59,632 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:54:59,632 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:54:59,632 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,632 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c714ec82d888461d8d7d28bcf4d59156, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2f941fffe7fb46188dac6c2bc393255e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/088aa43e292a4460834a3dc5504df986, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6d6aec2731474295828d31c5a683d1b0] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=48.3 K 2023-04-03 02:54:59,633 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:54:59,633 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c714ec82d888461d8d7d28bcf4d59156, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680490496049 2023-04-03 02:54:59,633 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:54:59,633 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,633 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/142a7e82795244a7a6976ccd4482c505, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7e70cff08f754ac2a247a78d87d7d199, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/76436b28b8e54fff88f9ef510cb78a10, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d5a384bed07f492abe1d99b1925aaca4] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=48.3 K 2023-04-03 02:54:59,633 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 2f941fffe7fb46188dac6c2bc393255e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680490496401 2023-04-03 02:54:59,634 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 142a7e82795244a7a6976ccd4482c505, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680490496049 2023-04-03 02:54:59,634 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 088aa43e292a4460834a3dc5504df986, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680490497507 2023-04-03 02:54:59,635 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7e70cff08f754ac2a247a78d87d7d199, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680490496401 2023-04-03 02:54:59,635 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6d6aec2731474295828d31c5a683d1b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680490498543 2023-04-03 02:54:59,636 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 76436b28b8e54fff88f9ef510cb78a10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680490497507 2023-04-03 02:54:59,637 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d5a384bed07f492abe1d99b1925aaca4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680490498543 2023-04-03 02:54:59,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:54:59,645 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:54:59,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:54:59,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:59,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:54:59,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:59,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:54:59,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:54:59,649 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:59,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:59,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:59,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:59,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:59,664 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:59,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:59,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:59,666 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:59,666 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#262 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:59,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,668 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,668 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:59,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:59,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:59,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:59,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:59,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:59,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:59,692 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:59,692 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:59,692 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:59,692 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:59,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:59,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:59,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:59,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490559691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,697 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:59,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c62b0a5[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:59,699 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:59,699 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490559697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,701 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:59,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:59,701 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,701 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:59,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:59,703 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:59,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:59,705 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:59,705 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:59,706 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:59,706 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:59,706 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:59,707 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,707 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,707 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,708 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:59,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,712 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,713 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,713 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,713 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:59,713 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:59,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,716 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/53b9bbb3a4e14ea78524126fbaf37dbe 2023-04-03 02:54:59,724 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/b3a42467107f47948d589ecea870418a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b3a42467107f47948d589ecea870418a 2023-04-03 02:54:59,730 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into b3a42467107f47948d589ecea870418a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:59,731 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:59,731 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=12, startTime=1680490499630; duration=0sec 2023-04-03 02:54:59,731 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/0241ba6d0c6d4d31acce5550754cf4e8 2023-04-03 02:54:59,731 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:54:59,731 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:54:59,732 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:54:59,733 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:54:59,733 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,733 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/15e9a48e5f3a49639bc26be87772b564, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/360dc7073b7e4a5381f6124aab180189, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/ddbf48134e7e4cbfb1d99bd9629bae73, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f5a640372b174939a327a4bde6fe09ac] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=48.3 K 2023-04-03 02:54:59,733 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 15e9a48e5f3a49639bc26be87772b564, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680490496049 2023-04-03 02:54:59,733 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 360dc7073b7e4a5381f6124aab180189, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680490496401 2023-04-03 02:54:59,734 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ddbf48134e7e4cbfb1d99bd9629bae73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680490497507 2023-04-03 02:54:59,735 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f5a640372b174939a327a4bde6fe09ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680490498543 2023-04-03 02:54:59,745 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#266 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:54:59,753 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/8e39a052981348ac8e64965f8d080409 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8e39a052981348ac8e64965f8d080409 2023-04-03 02:54:59,758 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into 8e39a052981348ac8e64965f8d080409(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:54:59,758 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:54:59,758 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=12, startTime=1680490499631; duration=0sec 2023-04-03 02:54:59,759 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:54:59,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490559782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490559783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490559784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490559797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:54:59,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490559801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,813 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:59,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:59,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:59,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:59,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:59,816 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:59,816 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,817 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:59,817 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:59,817 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,817 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,822 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,822 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,823 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,823 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:59,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:59,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:59,825 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:59,825 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,826 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:59,826 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:59,826 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:59,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:59,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:59,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:59,828 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,828 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,828 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:59,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c56b82[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:59,831 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:59,831 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:59,834 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,834 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:59,834 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:59,834 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:59,840 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:59,840 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,840 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:59,840 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:59,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:59,842 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:59,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:59,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:59,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:59,842 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:59,845 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,845 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,845 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,846 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,847 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,850 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,850 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,850 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,850 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,850 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:59,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:59,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:54:59,850 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,851 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:59,850 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,851 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:54:59,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,951 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:54:59,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:54:59,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:54:59,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:54:59,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:54:59,954 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:54:59,954 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,954 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:54:59,954 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:54:59,955 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,955 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,958 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,959 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,959 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:54:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:54:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:54:59,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:54:59,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:54:59,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:54:59,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:54:59,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:54:59,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:54:59,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:54:59,961 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,961 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:54:59,961 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:54:59,961 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:54:59,961 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:54:59,961 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:54:59,961 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,961 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:54:59,961 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:54:59,963 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,963 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,963 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,963 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:54:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:54:59,965 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:54:59,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b7829a4[Count = 0] remaining members to acquire global barrier 2023-04-03 02:54:59,965 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,966 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:54:59,966 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:54:59,966 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:54:59,967 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,967 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:54:59,967 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:54:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:54:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:54:59,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:54:59,970 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:54:59,970 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:54:59,970 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:54:59,970 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:54:59,970 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:54:59,971 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,971 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,971 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:54:59,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,972 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:54:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:54:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,973 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:54:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:54:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,978 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:54:59,978 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:54:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:54:59,978 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:54:59,978 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,978 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:54:59,978 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:54:59,978 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,978 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:54:59,978 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:54:59,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:54:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:54:59,979 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:54:59,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:54:59,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:54:59,979 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:54:59,980 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:00,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490560001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490560005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,080 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:00,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:00,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:00,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:00,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:00,083 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:00,083 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:00,083 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:00,083 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:00,083 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:00,083 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:00,084 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,084 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:00,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:00,085 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,085 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:00,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:00,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:00,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:00,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:00,086 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:00,086 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:00,086 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:00,086 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:00,088 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:00,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:00,089 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:00,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ae042f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:00,089 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:00,091 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:00,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:00,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:00,091 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,091 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:00,091 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:00,091 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:00,091 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:00,094 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:00,094 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:00,094 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,094 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:00,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:00,096 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:00,096 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:00,096 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:00,096 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:00,097 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:00,098 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:00,098 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:00,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:00,098 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:00,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:00,098 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,100 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:00,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:00,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:00,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:00,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:00,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:00,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:00,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:00,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:00,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:00,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:00,102 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:00,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:00,103 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:00,123 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/eaf1dd9c7f1f4b93ac92f163a42371af as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eaf1dd9c7f1f4b93ac92f163a42371af 2023-04-03 02:55:00,127 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into eaf1dd9c7f1f4b93ac92f163a42371af(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:00,128 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:00,128 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=12, startTime=1680490499630; duration=0sec 2023-04-03 02:55:00,128 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:00,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f17266d17a454603bb818386e274810d 2023-04-03 02:55:00,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/53b9bbb3a4e14ea78524126fbaf37dbe as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/53b9bbb3a4e14ea78524126fbaf37dbe 2023-04-03 02:55:00,155 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/53b9bbb3a4e14ea78524126fbaf37dbe, entries=150, sequenceid=364, filesize=11.9 K 2023-04-03 02:55:00,156 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/0241ba6d0c6d4d31acce5550754cf4e8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0241ba6d0c6d4d31acce5550754cf4e8 2023-04-03 02:55:00,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0241ba6d0c6d4d31acce5550754cf4e8, entries=150, sequenceid=364, filesize=11.9 K 2023-04-03 02:55:00,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f17266d17a454603bb818386e274810d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f17266d17a454603bb818386e274810d 2023-04-03 02:55:00,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f17266d17a454603bb818386e274810d, entries=150, sequenceid=364, filesize=11.9 K 2023-04-03 02:55:00,173 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ca59c59c9ebb324c8ea1a6deda6d6369 in 528ms, sequenceid=364, compaction requested=false 2023-04-03 02:55:00,173 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:00,203 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:00,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:00,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:00,206 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:00,206 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:00,206 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:00,206 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:00,206 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:00,206 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:00,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:00,208 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:00,208 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:00,208 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:00,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:00,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:00,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:00,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:00,209 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:00,210 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:00,210 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:00,210 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:00,210 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:00,210 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:00,210 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:00,210 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 02:55:00,210 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:00,211 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:00,211 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:00,211 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:00,211 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:00,211 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:00,288 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:00,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490560295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490560295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490560298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490560303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490560310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490560399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490560400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490560401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490560602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490560604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490560604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,626 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/9e08bb9f6eba4fbcbbc11eaa37d150a9 2023-04-03 02:55:00,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490560811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490560817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490560905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490560908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:00,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:00,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490560910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,040 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/4d4fbf8bb05e4c229708cbf0fe7b35a3 2023-04-03 02:55:01,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:01,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490561413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:01,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490561415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:01,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490561416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,452 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/da3d3951b4234af3b96ba92da09a154e 2023-04-03 02:55:01,456 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/9e08bb9f6eba4fbcbbc11eaa37d150a9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9e08bb9f6eba4fbcbbc11eaa37d150a9 2023-04-03 02:55:01,460 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9e08bb9f6eba4fbcbbc11eaa37d150a9, entries=150, sequenceid=387, filesize=11.9 K 2023-04-03 02:55:01,460 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/4d4fbf8bb05e4c229708cbf0fe7b35a3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4d4fbf8bb05e4c229708cbf0fe7b35a3 2023-04-03 02:55:01,464 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4d4fbf8bb05e4c229708cbf0fe7b35a3, entries=150, sequenceid=387, filesize=11.9 K 2023-04-03 02:55:01,464 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/da3d3951b4234af3b96ba92da09a154e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/da3d3951b4234af3b96ba92da09a154e 2023-04-03 02:55:01,468 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/da3d3951b4234af3b96ba92da09a154e, entries=150, sequenceid=387, filesize=11.9 K 2023-04-03 02:55:01,469 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ca59c59c9ebb324c8ea1a6deda6d6369 in 1259ms, sequenceid=387, compaction requested=true 2023-04-03 02:55:01,469 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:01,469 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:01,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:01,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:01,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:01,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:01,475 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,475 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:01,476 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:01,476 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:01,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:01,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:01,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:01,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0061c7[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:01,478 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:01,479 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:01,480 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,480 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:01,480 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:01,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:01,480 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:01,480 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:01,480 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:01,482 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:01,482 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:01,482 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,482 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:01,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:01,486 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:01,486 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:01,486 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:01,486 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:01,486 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:01,487 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:01,488 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:01,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:01,488 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:01,488 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:01,490 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:01,493 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:01,493 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:01,493 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:01,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:01,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:01,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:01,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:01,493 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:01,494 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:01,494 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:01,494 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:01,494 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,494 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:01,494 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:01,494 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:01,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:01,594 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:01,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:01,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:01,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:01,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:01,612 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:01,612 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:01,612 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:01,612 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:01,613 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:01,613 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:01,614 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:01,614 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:01,615 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,615 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:01,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:01,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:01,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:01,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:01,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:01,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:01,616 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:01,616 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:01,616 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:01,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:01,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:01,617 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:01,617 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:01,617 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:55:01,618 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:01,618 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:01,618 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:01,618 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:01,618 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:01,618 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:01,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:01,823 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:01,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:01,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490561847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:01,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490561849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:01,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490561950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:01,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:01,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490561952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,035 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/eb2e0d6d66a8423f8d29695ae7d5c578 2023-04-03 02:55:02,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:02,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490562152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:02,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490562155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:02,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490562419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:02,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490562420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:02,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490562426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,454 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/6f841e787f95452dbfac980f1b5e4f4b 2023-04-03 02:55:02,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:02,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490562457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:02,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490562457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,864 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/9ef434d09075496eb7b2afdc69f62b93 2023-04-03 02:55:02,868 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/eb2e0d6d66a8423f8d29695ae7d5c578 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eb2e0d6d66a8423f8d29695ae7d5c578 2023-04-03 02:55:02,872 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eb2e0d6d66a8423f8d29695ae7d5c578, entries=150, sequenceid=399, filesize=11.9 K 2023-04-03 02:55:02,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/6f841e787f95452dbfac980f1b5e4f4b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6f841e787f95452dbfac980f1b5e4f4b 2023-04-03 02:55:02,876 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6f841e787f95452dbfac980f1b5e4f4b, entries=150, sequenceid=399, filesize=11.9 K 2023-04-03 02:55:02,876 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/9ef434d09075496eb7b2afdc69f62b93 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ef434d09075496eb7b2afdc69f62b93 2023-04-03 02:55:02,880 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ef434d09075496eb7b2afdc69f62b93, entries=150, sequenceid=399, filesize=11.9 K 2023-04-03 02:55:02,880 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ca59c59c9ebb324c8ea1a6deda6d6369 in 1263ms, sequenceid=399, compaction requested=true 2023-04-03 02:55:02,880 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:02,881 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:02,881 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:02,881 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:02,881 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,881 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:02,881 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:02,889 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:02,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:02,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@262090de[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:02,891 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:02,891 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:02,893 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:02,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:02,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:02,893 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:02,893 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:02,893 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:02,893 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,893 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:02,894 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,894 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:02,894 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:02,894 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:02,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:02,897 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:02,897 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:02,897 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:02,897 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:02,897 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:02,899 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:02,899 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:02,899 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:02,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:02,900 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:02,901 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:02,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:02,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:02,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:02,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:02,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:02,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:02,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:02,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:02,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:02,903 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:02,903 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:02,903 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:02,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:02,903 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:02,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:02,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:02,961 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-03 02:55:02,961 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:02,961 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:02,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:02,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:02,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:02,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:02,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/5f3112bc838143c69b4e859eff0e7131 2023-04-03 02:55:02,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:02,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490562978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:02,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490562979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:02,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/c61798973a7c4a70b3684e4f1fe38fdd 2023-04-03 02:55:03,003 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:03,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:03,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:03,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:03,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:03,015 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:03,015 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:03,015 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:03,015 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:03,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,017 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f506142ad46549ef800af7ed37ea9738 2023-04-03 02:55:03,018 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,019 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,019 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:03,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:03,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:03,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:03,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:03,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:03,023 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:03,023 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:03,023 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:03,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/5f3112bc838143c69b4e859eff0e7131 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/5f3112bc838143c69b4e859eff0e7131 2023-04-03 02:55:03,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/5f3112bc838143c69b4e859eff0e7131, entries=200, sequenceid=420, filesize=14.3 K 2023-04-03 02:55:03,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/c61798973a7c4a70b3684e4f1fe38fdd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c61798973a7c4a70b3684e4f1fe38fdd 2023-04-03 02:55:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:03,029 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,029 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:03,029 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:03,029 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:03,031 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,031 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,032 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,032 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:03,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cdfa8b7[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:03,034 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:03,034 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,035 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,035 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:03,035 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:03,035 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:03,035 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,035 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:03,037 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:03,037 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:03,037 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:03,037 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:03,041 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c61798973a7c4a70b3684e4f1fe38fdd, entries=150, sequenceid=420, filesize=11.9 K 2023-04-03 02:55:03,041 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:03,041 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:03,041 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:03,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:03,041 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:03,041 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:03,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f506142ad46549ef800af7ed37ea9738 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f506142ad46549ef800af7ed37ea9738 2023-04-03 02:55:03,046 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f506142ad46549ef800af7ed37ea9738, entries=150, sequenceid=420, filesize=11.9 K 2023-04-03 02:55:03,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for ca59c59c9ebb324c8ea1a6deda6d6369 in 85ms, sequenceid=420, compaction requested=true 2023-04-03 02:55:03,046 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,047 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,047 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:03,047 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:55:03,047 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,047 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:03,047 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:55:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,047 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:03,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,048 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64188 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:55:03,049 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:55:03,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,049 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,049 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eaf1dd9c7f1f4b93ac92f163a42371af, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/53b9bbb3a4e14ea78524126fbaf37dbe, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9e08bb9f6eba4fbcbbc11eaa37d150a9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eb2e0d6d66a8423f8d29695ae7d5c578, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/5f3112bc838143c69b4e859eff0e7131] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=62.7 K 2023-04-03 02:55:03,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,049 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,050 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:55:03,050 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:55:03,050 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,050 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b3a42467107f47948d589ecea870418a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0241ba6d0c6d4d31acce5550754cf4e8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4d4fbf8bb05e4c229708cbf0fe7b35a3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6f841e787f95452dbfac980f1b5e4f4b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c61798973a7c4a70b3684e4f1fe38fdd] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=60.3 K 2023-04-03 02:55:03,051 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting eaf1dd9c7f1f4b93ac92f163a42371af, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680490498543 2023-04-03 02:55:03,051 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b3a42467107f47948d589ecea870418a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680490498543 2023-04-03 02:55:03,051 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 53b9bbb3a4e14ea78524126fbaf37dbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680490499160 2023-04-03 02:55:03,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,052 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 0241ba6d0c6d4d31acce5550754cf4e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680490499160 2023-04-03 02:55:03,052 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 9e08bb9f6eba4fbcbbc11eaa37d150a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680490499669 2023-04-03 02:55:03,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,053 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 4d4fbf8bb05e4c229708cbf0fe7b35a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680490499669 2023-04-03 02:55:03,053 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting eb2e0d6d66a8423f8d29695ae7d5c578, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1680490500290 2023-04-03 02:55:03,053 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 6f841e787f95452dbfac980f1b5e4f4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1680490500290 2023-04-03 02:55:03,053 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 5f3112bc838143c69b4e859eff0e7131, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680490501843 2023-04-03 02:55:03,054 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting c61798973a7c4a70b3684e4f1fe38fdd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680490501843 2023-04-03 02:55:03,054 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:03,054 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:03,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:03,054 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,054 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,054 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,054 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:03,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:03,054 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,054 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,054 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,055 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:03,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:03,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:03,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:03,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:03,055 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:03,055 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:03,071 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#276 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:03,082 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#277 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:03,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:03,085 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:55:03,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:03,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:03,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:03,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:03,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:03,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:03,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/60edeec55c264fafbdeeae74bc47851b 2023-04-03 02:55:03,111 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/b174603456a34e36930b26da989ee291 2023-04-03 02:55:03,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:03,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490563118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:03,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490563121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,155 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:03,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:03,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:03,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:03,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:03,159 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:03,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:03,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:03,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:03,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,161 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:03,161 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:03,161 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,161 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:03,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:03,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:03,162 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,162 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:03,162 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:03,163 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,163 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:03,163 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:03,163 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,163 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:03,163 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:03,164 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,164 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,165 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,165 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af6467f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:03,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:03,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,168 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:03,169 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,169 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:03,170 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:03,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:03,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:03,172 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:03,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:03,173 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:03,173 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:03,173 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:03,174 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,174 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,174 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:03,175 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:03,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:03,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:03,178 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:03,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:03,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:03,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:03,179 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:03,179 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:03,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:03,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490563221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:03,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490563224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,279 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:03,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:03,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:03,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:03,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:03,282 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:03,282 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:03,282 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:03,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:03,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,285 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:03,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:03,285 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,285 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:03,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:03,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:03,287 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,287 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:03,287 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:03,287 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:03,289 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,289 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,289 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,289 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:03,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@566418bf[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:03,291 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:03,291 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,292 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:03,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:03,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:03,292 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,292 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:03,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,301 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:03,301 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,301 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:03,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:03,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:03,304 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:03,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:03,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:03,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:03,304 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:03,305 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,305 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,305 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:03,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:03,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:03,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,307 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:03,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:03,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:03,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:03,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:03,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:03,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:03,310 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:03,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,310 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:03,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,310 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:03,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:03,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:03,411 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:03,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:03,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:03,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:03,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:03,414 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:03,414 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:03,414 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:03,414 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:03,414 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,414 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,416 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,416 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:03,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:03,416 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,416 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:03,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:03,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:03,418 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,418 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:03,418 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:03,418 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,418 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:03,418 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:03,418 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,418 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:03,418 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:03,420 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:03,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e0c4cd[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:03,422 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:03,422 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,423 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,423 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:03,423 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,423 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:03,423 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:03,423 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:03,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,425 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,425 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:03,425 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:03,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:03,427 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:03,427 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:03,427 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:03,427 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:03,427 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:03,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:03,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490563425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:03,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:03,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:03,429 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:03,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490563427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,431 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,434 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,434 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:03,434 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:03,435 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:03,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:03,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:03,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:03,435 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:03,435 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,435 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:03,435 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:03,435 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,435 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:03,435 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,497 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/ac40fb1251e94acb8079874d09505110 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/ac40fb1251e94acb8079874d09505110 2023-04-03 02:55:03,500 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into ac40fb1251e94acb8079874d09505110(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:03,501 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:03,501 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=11, startTime=1680490503047; duration=0sec 2023-04-03 02:55:03,501 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:03,501 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:55:03,502 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:55:03,502 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:55:03,502 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,502 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8e39a052981348ac8e64965f8d080409, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f17266d17a454603bb818386e274810d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/da3d3951b4234af3b96ba92da09a154e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ef434d09075496eb7b2afdc69f62b93, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f506142ad46549ef800af7ed37ea9738] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=60.3 K 2023-04-03 02:55:03,502 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 8e39a052981348ac8e64965f8d080409, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680490498543 2023-04-03 02:55:03,503 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f17266d17a454603bb818386e274810d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680490499160 2023-04-03 02:55:03,503 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting da3d3951b4234af3b96ba92da09a154e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680490499669 2023-04-03 02:55:03,504 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 9ef434d09075496eb7b2afdc69f62b93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1680490500290 2023-04-03 02:55:03,504 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f506142ad46549ef800af7ed37ea9738, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680490501843 2023-04-03 02:55:03,507 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/700f20793c534a9fa9861e64ca6a0936 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/700f20793c534a9fa9861e64ca6a0936 2023-04-03 02:55:03,512 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into 700f20793c534a9fa9861e64ca6a0936(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:03,512 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:03,512 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=11, startTime=1680490503047; duration=0sec 2023-04-03 02:55:03,512 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:03,513 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:03,521 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/242e30fadd724f489088286c01c7ef60 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/242e30fadd724f489088286c01c7ef60 2023-04-03 02:55:03,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/e403b02e4e3f425794b3f27a786f4af1 2023-04-03 02:55:03,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/60edeec55c264fafbdeeae74bc47851b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60edeec55c264fafbdeeae74bc47851b 2023-04-03 02:55:03,527 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into 242e30fadd724f489088286c01c7ef60(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:03,527 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:03,528 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=11, startTime=1680490503047; duration=0sec 2023-04-03 02:55:03,528 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:03,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60edeec55c264fafbdeeae74bc47851b, entries=200, sequenceid=432, filesize=14.3 K 2023-04-03 02:55:03,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/b174603456a34e36930b26da989ee291 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b174603456a34e36930b26da989ee291 2023-04-03 02:55:03,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b174603456a34e36930b26da989ee291, entries=150, sequenceid=432, filesize=11.9 K 2023-04-03 02:55:03,535 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:03,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/e403b02e4e3f425794b3f27a786f4af1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e403b02e4e3f425794b3f27a786f4af1 2023-04-03 02:55:03,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e403b02e4e3f425794b3f27a786f4af1, entries=150, sequenceid=432, filesize=11.9 K 2023-04-03 02:55:03,540 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ca59c59c9ebb324c8ea1a6deda6d6369 in 455ms, sequenceid=432, compaction requested=false 2023-04-03 02:55:03,540 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:03,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:03,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:03,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:03,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:03,544 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:03,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:03,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:03,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:03,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:03,545 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,546 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,546 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:03,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:03,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:03,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:03,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:03,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:03,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:03,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:03,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:03,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:03,548 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:03,548 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:03,548 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 02:55:03,549 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:03,549 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:03,549 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:03,549 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:03,549 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:03,549 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:03,568 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/eac2135b06d241c4b2130f64f55fc802 2023-04-03 02:55:03,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:03,732 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:03,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:03,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:03,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490563748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490563749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:03,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490563853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:03,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490563853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:03,985 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/a5b3b76c61b04bb696a372814202584a 2023-04-03 02:55:03,995 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/d588af922f7f422883cc9a357688fb8b 2023-04-03 02:55:03,999 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/eac2135b06d241c4b2130f64f55fc802 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eac2135b06d241c4b2130f64f55fc802 2023-04-03 02:55:04,002 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eac2135b06d241c4b2130f64f55fc802, entries=150, sequenceid=455, filesize=11.9 K 2023-04-03 02:55:04,004 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/a5b3b76c61b04bb696a372814202584a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/a5b3b76c61b04bb696a372814202584a 2023-04-03 02:55:04,008 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/a5b3b76c61b04bb696a372814202584a, entries=150, sequenceid=455, filesize=11.9 K 2023-04-03 02:55:04,009 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/d588af922f7f422883cc9a357688fb8b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/d588af922f7f422883cc9a357688fb8b 2023-04-03 02:55:04,012 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/d588af922f7f422883cc9a357688fb8b, entries=150, sequenceid=455, filesize=11.9 K 2023-04-03 02:55:04,013 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for ca59c59c9ebb324c8ea1a6deda6d6369 in 465ms, sequenceid=455, compaction requested=true 2023-04-03 02:55:04,013 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:04,013 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:04,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,016 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:04,016 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:04,018 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,018 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,018 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,018 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:04,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:04,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@656b50f6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:04,020 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:04,020 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,022 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,023 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:04,023 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:04,023 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:04,023 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,023 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:04,026 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,026 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:04,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,026 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,027 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:04,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:04,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:04,031 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:04,031 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:04,031 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:04,031 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:04,031 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:04,033 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,034 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,034 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,035 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,037 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,040 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,040 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,040 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,040 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,040 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,040 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:04,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:04,040 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,041 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,041 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,041 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:04,041 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:04,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:04,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:04,057 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 02:55:04,057 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:04,057 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:04,057 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:04,057 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:04,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:04,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:04,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/06643d40c4d54cb1b6733cdc16b3abb4 2023-04-03 02:55:04,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490564081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490564082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,106 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/577b6b46ad4c4042b7466e3aba241654 2023-04-03 02:55:04,141 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:04,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:04,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:04,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:04,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:04,146 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:04,146 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,146 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:04,146 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:04,146 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,147 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,148 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,149 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,149 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,149 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:04,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:04,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:04,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:04,151 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:04,151 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,151 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:04,151 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:04,151 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:04,151 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,151 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:04,151 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:04,152 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:04,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:04,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:04,152 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:04,154 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,154 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,154 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,154 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:04,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d56379a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:04,158 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:04,158 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:04,160 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:04,160 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:04,162 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:04,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:04,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:04,165 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:04,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:04,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:04,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:04,166 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:04,171 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,171 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,171 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,171 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,173 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:04,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:04,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:04,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,177 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,177 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,177 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:04,177 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,177 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:04,177 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490564184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490564186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,277 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:04,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:04,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:04,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:04,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:04,280 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:04,280 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,280 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:04,280 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:04,281 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,281 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,282 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,282 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,282 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,282 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:04,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:04,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:04,283 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:04,284 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,284 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:04,284 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:04,284 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:04,286 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,286 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,286 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,286 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:04,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b01756f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:04,288 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:04,288 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,290 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,290 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,290 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:04,290 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:04,290 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:04,290 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:04,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:04,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,292 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:04,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:04,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:04,295 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:04,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:04,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:04,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:04,295 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:04,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,297 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,298 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,301 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:04,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,301 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:04,301 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:04,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:04,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490564387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490564391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,401 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:04,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:04,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:04,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:04,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:04,405 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:04,405 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,405 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:04,405 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:04,405 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,405 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,408 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,408 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,408 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,408 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:04,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:04,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:04,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:04,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:04,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:04,410 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:04,410 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,410 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:04,410 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:04,410 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,410 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:04,410 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:04,410 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,410 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:04,410 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:04,411 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,411 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,412 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,412 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:04,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1935026e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:04,413 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:04,413 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,415 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,415 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:04,415 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:04,415 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:04,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:04,416 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:04,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:04,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:04,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:04,418 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:04,418 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:04,418 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:04,418 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:04,419 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:04,421 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,421 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,421 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,422 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,423 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,425 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:04,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:04,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,425 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:04,426 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:04,426 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,426 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,426 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,426 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490564435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,437 DEBUG [Thread-1526] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:04,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490564435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,438 DEBUG [Thread-1532] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:04,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490564441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,443 DEBUG [Thread-1524] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:04,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/1e1ce09dc0954bf5bd30ccf6ad3f3e04 2023-04-03 02:55:04,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/06643d40c4d54cb1b6733cdc16b3abb4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/06643d40c4d54cb1b6733cdc16b3abb4 2023-04-03 02:55:04,526 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:04,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:04,527 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/06643d40c4d54cb1b6733cdc16b3abb4, entries=150, sequenceid=470, filesize=11.9 K 2023-04-03 02:55:04,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/577b6b46ad4c4042b7466e3aba241654 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/577b6b46ad4c4042b7466e3aba241654 2023-04-03 02:55:04,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:04,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:04,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:04,529 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:04,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:04,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:04,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/577b6b46ad4c4042b7466e3aba241654, entries=150, sequenceid=470, filesize=11.9 K 2023-04-03 02:55:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:04,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:04,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/1e1ce09dc0954bf5bd30ccf6ad3f3e04 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1e1ce09dc0954bf5bd30ccf6ad3f3e04 2023-04-03 02:55:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:04,533 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,533 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:04,533 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:04,533 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:04,535 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,535 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,535 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,535 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1e1ce09dc0954bf5bd30ccf6ad3f3e04, entries=150, sequenceid=470, filesize=11.9 K 2023-04-03 02:55:04,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ca59c59c9ebb324c8ea1a6deda6d6369 in 480ms, sequenceid=470, compaction requested=true 2023-04-03 02:55:04,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:04,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:04,537 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:55:04,538 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:55:04,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:04,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:04,539 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52172 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:55:04,539 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:55:04,539 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:55:04,539 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:55:04,539 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,539 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,539 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/700f20793c534a9fa9861e64ca6a0936, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60edeec55c264fafbdeeae74bc47851b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eac2135b06d241c4b2130f64f55fc802, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/06643d40c4d54cb1b6733cdc16b3abb4] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=50.9 K 2023-04-03 02:55:04,539 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/ac40fb1251e94acb8079874d09505110, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b174603456a34e36930b26da989ee291, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/a5b3b76c61b04bb696a372814202584a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/577b6b46ad4c4042b7466e3aba241654] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=48.6 K 2023-04-03 02:55:04,539 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 700f20793c534a9fa9861e64ca6a0936, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680490501843 2023-04-03 02:55:04,539 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ac40fb1251e94acb8079874d09505110, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680490501843 2023-04-03 02:55:04,540 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 60edeec55c264fafbdeeae74bc47851b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680490502974 2023-04-03 02:55:04,540 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b174603456a34e36930b26da989ee291, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680490502974 2023-04-03 02:55:04,540 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting eac2135b06d241c4b2130f64f55fc802, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680490503115 2023-04-03 02:55:04,540 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a5b3b76c61b04bb696a372814202584a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680490503115 2023-04-03 02:55:04,541 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 06643d40c4d54cb1b6733cdc16b3abb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680490503743 2023-04-03 02:55:04,541 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 577b6b46ad4c4042b7466e3aba241654, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680490503743 2023-04-03 02:55:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:04,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26660d5e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:04,551 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:04,552 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,553 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#288 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:04,554 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#289 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:04,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:04,555 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,555 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:04,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:04,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:04,558 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:04,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:04,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:04,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:04,559 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:04,560 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,560 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,560 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,562 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,569 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,569 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,569 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,569 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:04,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:04,569 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,569 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:04,569 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,569 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,569 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,569 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,570 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:04,570 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:04,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,573 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/70eefa62ea5d4d71a5808bc9bee77857 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/70eefa62ea5d4d71a5808bc9bee77857 2023-04-03 02:55:04,577 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into 70eefa62ea5d4d71a5808bc9bee77857(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:04,577 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:04,577 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=12, startTime=1680490504537; duration=0sec 2023-04-03 02:55:04,577 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:04,578 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:55:04,579 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:55:04,579 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:55:04,579 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,579 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/242e30fadd724f489088286c01c7ef60, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e403b02e4e3f425794b3f27a786f4af1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/d588af922f7f422883cc9a357688fb8b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1e1ce09dc0954bf5bd30ccf6ad3f3e04] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=48.6 K 2023-04-03 02:55:04,579 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 242e30fadd724f489088286c01c7ef60, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680490501843 2023-04-03 02:55:04,579 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e403b02e4e3f425794b3f27a786f4af1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680490502974 2023-04-03 02:55:04,580 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d588af922f7f422883cc9a357688fb8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680490503115 2023-04-03 02:55:04,580 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 1e1ce09dc0954bf5bd30ccf6ad3f3e04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680490503743 2023-04-03 02:55:04,587 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#290 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:04,596 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/886ccf3bda82444c9c86c16328cb7858 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/886ccf3bda82444c9c86c16328cb7858 2023-04-03 02:55:04,602 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into 886ccf3bda82444c9c86c16328cb7858(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:04,602 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:04,602 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=12, startTime=1680490504538; duration=0sec 2023-04-03 02:55:04,602 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:04,670 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:04,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:04,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:04,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:04,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:04,673 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:04,673 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,673 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:04,673 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:04,674 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,674 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,675 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,675 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,676 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,676 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:04,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:04,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:04,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:04,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:04,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:04,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:04,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:04,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:04,677 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,677 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:04,678 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 02:55:04,678 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:04,678 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:04,678 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:04,678 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:04,678 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:04,678 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:04,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:04,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:04,701 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/dce52598bd5c4af39b72ac7b4a73376e 2023-04-03 02:55:04,723 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/52a188bb271842f2a3ab1d6b1d02d155 2023-04-03 02:55:04,738 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f028ce727ff44ca592e1254492448634 2023-04-03 02:55:04,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490564735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490564739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,759 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/dce52598bd5c4af39b72ac7b4a73376e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/dce52598bd5c4af39b72ac7b4a73376e 2023-04-03 02:55:04,770 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/dce52598bd5c4af39b72ac7b4a73376e, entries=150, sequenceid=489, filesize=11.9 K 2023-04-03 02:55:04,771 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/52a188bb271842f2a3ab1d6b1d02d155 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/52a188bb271842f2a3ab1d6b1d02d155 2023-04-03 02:55:04,775 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/52a188bb271842f2a3ab1d6b1d02d155, entries=150, sequenceid=489, filesize=11.9 K 2023-04-03 02:55:04,776 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f028ce727ff44ca592e1254492448634 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f028ce727ff44ca592e1254492448634 2023-04-03 02:55:04,780 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f028ce727ff44ca592e1254492448634, entries=150, sequenceid=489, filesize=11.9 K 2023-04-03 02:55:04,781 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ca59c59c9ebb324c8ea1a6deda6d6369 in 103ms, sequenceid=489, compaction requested=false 2023-04-03 02:55:04,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:04,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:04,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:04,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,792 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:04,792 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:04,793 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,793 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,794 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,794 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:04,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2014ef[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:04,796 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:04,796 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:04,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:04,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:04,799 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,799 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:04,802 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:04,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:04,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:04,807 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:04,807 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:04,807 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:04,807 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:04,807 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:04,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,812 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,815 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,819 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:04,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:04,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:04,820 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:04,820 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:04,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:04,843 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 02:55:04,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:04,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:04,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:04,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:04,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:04,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:04,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/562dbddbbef648f0b0e00eaff93dcdb2 2023-04-03 02:55:04,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490564871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490564873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,920 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:04,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:04,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:04,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:04,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:04,924 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:04,924 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,924 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:04,924 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:04,924 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/7d35d5adc73149f184bc53ba56bb488f 2023-04-03 02:55:04,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,926 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,926 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,927 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,927 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:04,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:04,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:04,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:04,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:04,936 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:04,936 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:04,936 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:04,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:04,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:04,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:04,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:04,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:04,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:04,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:04,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,956 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:04,959 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,959 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6ab2c9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:04,963 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:04,963 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,964 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:04,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:04,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:04,965 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,965 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:04,966 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:04,966 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,966 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:04,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:04,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:04,972 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:04,972 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:04,972 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:04,972 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:04,972 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:04,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/c2e12daa0cdd4ce1895dd7e12ea7d2d0 2023-04-03 02:55:04,976 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:04,976 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,976 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:04,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,977 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,978 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/b10a4a61eb904c72b341a97a814f9cf3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b10a4a61eb904c72b341a97a814f9cf3 2023-04-03 02:55:04,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:04,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490564976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:04,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490564976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:04,981 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:04,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/562dbddbbef648f0b0e00eaff93dcdb2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/562dbddbbef648f0b0e00eaff93dcdb2 2023-04-03 02:55:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,990 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into b10a4a61eb904c72b341a97a814f9cf3(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:04,990 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:04,990 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=12, startTime=1680490504537; duration=0sec 2023-04-03 02:55:04,990 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:04,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/562dbddbbef648f0b0e00eaff93dcdb2, entries=200, sequenceid=505, filesize=14.3 K 2023-04-03 02:55:04,996 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,996 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,996 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:04,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:04,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:04,996 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:04,996 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:04,996 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,997 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:04,997 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:04,996 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:04,997 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:04,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:04,996 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:04,997 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:04,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:05,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/7d35d5adc73149f184bc53ba56bb488f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7d35d5adc73149f184bc53ba56bb488f 2023-04-03 02:55:05,021 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7d35d5adc73149f184bc53ba56bb488f, entries=150, sequenceid=505, filesize=11.9 K 2023-04-03 02:55:05,022 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/c2e12daa0cdd4ce1895dd7e12ea7d2d0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c2e12daa0cdd4ce1895dd7e12ea7d2d0 2023-04-03 02:55:05,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c2e12daa0cdd4ce1895dd7e12ea7d2d0, entries=150, sequenceid=505, filesize=11.9 K 2023-04-03 02:55:05,032 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for ca59c59c9ebb324c8ea1a6deda6d6369 in 189ms, sequenceid=505, compaction requested=true 2023-04-03 02:55:05,032 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,032 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:05,032 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:05,032 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:05,032 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:05,033 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:05,033 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:05,033 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:55:05,034 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,034 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/70eefa62ea5d4d71a5808bc9bee77857, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/52a188bb271842f2a3ab1d6b1d02d155, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7d35d5adc73149f184bc53ba56bb488f] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=36.8 K 2023-04-03 02:55:05,034 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:05,034 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 70eefa62ea5d4d71a5808bc9bee77857, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680490503743 2023-04-03 02:55:05,034 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:55:05,034 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,034 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b10a4a61eb904c72b341a97a814f9cf3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/dce52598bd5c4af39b72ac7b4a73376e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/562dbddbbef648f0b0e00eaff93dcdb2] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=39.2 K 2023-04-03 02:55:05,035 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 52a188bb271842f2a3ab1d6b1d02d155, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680490504076 2023-04-03 02:55:05,035 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting b10a4a61eb904c72b341a97a814f9cf3, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680490503743 2023-04-03 02:55:05,035 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7d35d5adc73149f184bc53ba56bb488f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680490504728 2023-04-03 02:55:05,035 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting dce52598bd5c4af39b72ac7b4a73376e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680490504076 2023-04-03 02:55:05,036 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 562dbddbbef648f0b0e00eaff93dcdb2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680490504728 2023-04-03 02:55:05,042 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#297 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:05,049 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#298 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:05,083 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/cd0d8fc76f0b44309912ae6bab2cc0ed as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cd0d8fc76f0b44309912ae6bab2cc0ed 2023-04-03 02:55:05,092 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into cd0d8fc76f0b44309912ae6bab2cc0ed(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:05,093 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,093 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=13, startTime=1680490505032; duration=0sec 2023-04-03 02:55:05,093 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:05,093 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:05,094 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:05,094 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:55:05,094 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,094 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/886ccf3bda82444c9c86c16328cb7858, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f028ce727ff44ca592e1254492448634, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c2e12daa0cdd4ce1895dd7e12ea7d2d0] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=36.8 K 2023-04-03 02:55:05,095 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 886ccf3bda82444c9c86c16328cb7858, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680490503743 2023-04-03 02:55:05,095 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting f028ce727ff44ca592e1254492448634, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680490504076 2023-04-03 02:55:05,096 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c2e12daa0cdd4ce1895dd7e12ea7d2d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680490504728 2023-04-03 02:55:05,097 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:05,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:05,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:05,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:05,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:05,108 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:05,108 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,108 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:05,108 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:05,109 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,109 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,109 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#299 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:05,110 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:05,110 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:05,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:05,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:05,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:05,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:05,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:05,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:05,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:05,120 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,120 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:05,120 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 02:55:05,120 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:05,120 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:05,121 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:05,121 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:05,121 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:05,121 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:05,130 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/9c338ec7481745c5a3b89437753af0e7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9c338ec7481745c5a3b89437753af0e7 2023-04-03 02:55:05,136 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into 9c338ec7481745c5a3b89437753af0e7(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:05,136 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,136 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=13, startTime=1680490505033; duration=0sec 2023-04-03 02:55:05,136 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:05,138 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/0a32b2ef97c64959b4dec90b3ebb1394 2023-04-03 02:55:05,149 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/cac489b1dd7d4eff96382dab41f31e24 2023-04-03 02:55:05,159 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/809786cd50bd40f79a0b3cf2735fc7fc 2023-04-03 02:55:05,165 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/0a32b2ef97c64959b4dec90b3ebb1394 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0a32b2ef97c64959b4dec90b3ebb1394 2023-04-03 02:55:05,168 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0a32b2ef97c64959b4dec90b3ebb1394, entries=150, sequenceid=523, filesize=11.9 K 2023-04-03 02:55:05,169 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/cac489b1dd7d4eff96382dab41f31e24 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cac489b1dd7d4eff96382dab41f31e24 2023-04-03 02:55:05,172 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cac489b1dd7d4eff96382dab41f31e24, entries=150, sequenceid=523, filesize=11.9 K 2023-04-03 02:55:05,173 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/809786cd50bd40f79a0b3cf2735fc7fc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/809786cd50bd40f79a0b3cf2735fc7fc 2023-04-03 02:55:05,177 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/809786cd50bd40f79a0b3cf2735fc7fc, entries=150, sequenceid=523, filesize=11.9 K 2023-04-03 02:55:05,178 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for ca59c59c9ebb324c8ea1a6deda6d6369 in 58ms, sequenceid=523, compaction requested=false 2023-04-03 02:55:05,178 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,178 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:05,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:05,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:05,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:05,180 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,181 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,181 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,181 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:05,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bfdfb98[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:05,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:05,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:05,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:05,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:05,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:05,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:05,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:05,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:05,191 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:05,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:05,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:05,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:05,192 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:05,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:05,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,194 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,195 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:05,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:55:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:05,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:05,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:05,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:05,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:05,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:05,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:05,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:05,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:05,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:05,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:05,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:05,200 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:05,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:05,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:05,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:05,201 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:05,201 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:05,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/ae993ab9052b466db9fe79e55eff5fc8 2023-04-03 02:55:05,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490505222, max timestamp=1680490505236Num uniques 859; 2023-04-03 02:55:05,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:05,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490505222, max timestamp=1680490505236Num uniques 859; 2023-04-03 02:55:05,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:05,250 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/4b66b8305991458794251db6eefb1b8d 2023-04-03 02:55:05,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490565254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490565256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,259 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/05858b1ae8fd4249b750d0e6a26c2716 2023-04-03 02:55:05,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/ae993ab9052b466db9fe79e55eff5fc8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/ae993ab9052b466db9fe79e55eff5fc8 2023-04-03 02:55:05,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/ae993ab9052b466db9fe79e55eff5fc8, entries=200, sequenceid=535, filesize=14.3 K 2023-04-03 02:55:05,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/4b66b8305991458794251db6eefb1b8d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4b66b8305991458794251db6eefb1b8d 2023-04-03 02:55:05,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4b66b8305991458794251db6eefb1b8d, entries=150, sequenceid=535, filesize=11.9 K 2023-04-03 02:55:05,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/05858b1ae8fd4249b750d0e6a26c2716 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/05858b1ae8fd4249b750d0e6a26c2716 2023-04-03 02:55:05,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/05858b1ae8fd4249b750d0e6a26c2716, entries=150, sequenceid=535, filesize=11.9 K 2023-04-03 02:55:05,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for ca59c59c9ebb324c8ea1a6deda6d6369 in 87ms, sequenceid=535, compaction requested=true 2023-04-03 02:55:05,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:05,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:05,283 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:05,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:55:05,284 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40224 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:05,284 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:55:05,284 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,284 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cd0d8fc76f0b44309912ae6bab2cc0ed, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0a32b2ef97c64959b4dec90b3ebb1394, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/ae993ab9052b466db9fe79e55eff5fc8] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=39.3 K 2023-04-03 02:55:05,285 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting cd0d8fc76f0b44309912ae6bab2cc0ed, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680490504728 2023-04-03 02:55:05,285 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 0a32b2ef97c64959b4dec90b3ebb1394, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1680490504865 2023-04-03 02:55:05,285 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting ae993ab9052b466db9fe79e55eff5fc8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1680490505190 2023-04-03 02:55:05,292 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:05,301 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:05,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:05,302 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/99fcf2a656bf463db3780f2549e5e89b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/99fcf2a656bf463db3780f2549e5e89b 2023-04-03 02:55:05,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:05,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:05,306 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into 99fcf2a656bf463db3780f2549e5e89b(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:05,307 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,307 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=13, startTime=1680490505283; duration=0sec 2023-04-03 02:55:05,307 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:55:05,307 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-03 02:55:05,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:05,307 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:05,307 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,307 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:05,307 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:05,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,308 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 02:55:05,308 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 02:55:05,308 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. because compaction request was cancelled 2023-04-03 02:55:05,308 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:05,309 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:05,309 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:55:05,309 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,309 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9c338ec7481745c5a3b89437753af0e7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/809786cd50bd40f79a0b3cf2735fc7fc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/05858b1ae8fd4249b750d0e6a26c2716] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=36.9 K 2023-04-03 02:55:05,310 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 9c338ec7481745c5a3b89437753af0e7, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680490504728 2023-04-03 02:55:05,310 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 809786cd50bd40f79a0b3cf2735fc7fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1680490504865 2023-04-03 02:55:05,310 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 05858b1ae8fd4249b750d0e6a26c2716, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1680490505190 2023-04-03 02:55:05,313 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,313 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:05,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:05,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:05,313 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,313 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:05,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:05,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:05,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:05,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:05,315 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:05,315 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,315 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:05,315 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:05,315 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:05,315 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,315 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:05,316 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=125.23 KB heapSize=363.39 KB 2023-04-03 02:55:05,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:05,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:05,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:05,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:05,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:05,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:05,325 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#307 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:05,352 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/22f85985e1d6421a91393d8eacda66bc 2023-04-03 02:55:05,354 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/601265f85f424658b8b66f1757c42875 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/601265f85f424658b8b66f1757c42875 2023-04-03 02:55:05,359 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into 601265f85f424658b8b66f1757c42875(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:05,360 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,360 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=13, startTime=1680490505283; duration=0sec 2023-04-03 02:55:05,360 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:05,361 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:05,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:05,373 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/f80f5b86c6a04496b6635113af3c57d8 2023-04-03 02:55:05,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490565383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490565384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,420 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/52abd4dfed1a4c36add13d700268a46d 2023-04-03 02:55:05,426 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/22f85985e1d6421a91393d8eacda66bc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/22f85985e1d6421a91393d8eacda66bc 2023-04-03 02:55:05,429 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/22f85985e1d6421a91393d8eacda66bc, entries=150, sequenceid=567, filesize=11.9 K 2023-04-03 02:55:05,430 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/f80f5b86c6a04496b6635113af3c57d8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/f80f5b86c6a04496b6635113af3c57d8 2023-04-03 02:55:05,433 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/f80f5b86c6a04496b6635113af3c57d8, entries=150, sequenceid=567, filesize=11.9 K 2023-04-03 02:55:05,434 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/52abd4dfed1a4c36add13d700268a46d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/52abd4dfed1a4c36add13d700268a46d 2023-04-03 02:55:05,442 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/52abd4dfed1a4c36add13d700268a46d, entries=150, sequenceid=567, filesize=11.9 K 2023-04-03 02:55:05,442 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~363.34 KB/372064, currentSize=53.67 KB/54960 for ca59c59c9ebb324c8ea1a6deda6d6369 in 127ms, sequenceid=567, compaction requested=true 2023-04-03 02:55:05,443 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,443 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,443 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:05,443 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:05,443 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,444 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:05,444 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:05,446 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,446 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,447 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,447 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:05,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d670144[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:05,450 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:05,451 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,453 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,453 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:05,453 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:05,453 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:05,453 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,453 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:05,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:05,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:05,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,455 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:05,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:05,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:05,458 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:05,459 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:05,459 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:05,459 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:05,459 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:05,462 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,462 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:05,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,462 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:05,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:05,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,463 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,465 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,468 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,468 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:05,468 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:05,468 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:05,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:05,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:05,468 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,469 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,469 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:05,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:05,469 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,469 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,469 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,469 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:05,469 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:05,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:05,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:05,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:05,480 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/bb373193ab834c828792b0d9b1f02eec as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/bb373193ab834c828792b0d9b1f02eec 2023-04-03 02:55:05,486 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into bb373193ab834c828792b0d9b1f02eec(size=13.1 K), total size for store is 48.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:05,486 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,487 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=13, startTime=1680490505032; duration=0sec 2023-04-03 02:55:05,487 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:05,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:05,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:55:05,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:05,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:05,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:05,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:05,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:05,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:05,507 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/8a2f0dff6eb24ff4a61887e7dd38aad6 2023-04-03 02:55:05,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490505519, max timestamp=1680490505532Num uniques 859; 2023-04-03 02:55:05,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:05,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490505519, max timestamp=1680490505532Num uniques 859; 2023-04-03 02:55:05,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:05,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490505519, max timestamp=1680490505532Num uniques 859; 2023-04-03 02:55:05,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:05,535 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/cf4f213735744522b2cfb2b88d9e2b68 2023-04-03 02:55:05,569 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:05,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490565566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:05,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:05,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490565569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:05,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:05,596 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:05,596 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/aa4300019a7a426c818742635b1b51d0 2023-04-03 02:55:05,596 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,596 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:05,596 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:05,597 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,597 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,598 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:05,598 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:05,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:05,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,599 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,599 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:05,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:05,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:05,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:05,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:05,600 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:05,600 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,600 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:05,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/8a2f0dff6eb24ff4a61887e7dd38aad6 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8a2f0dff6eb24ff4a61887e7dd38aad6 2023-04-03 02:55:05,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8a2f0dff6eb24ff4a61887e7dd38aad6, entries=150, sequenceid=581, filesize=11.9 K 2023-04-03 02:55:05,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/cf4f213735744522b2cfb2b88d9e2b68 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cf4f213735744522b2cfb2b88d9e2b68 2023-04-03 02:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:05,608 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:05,608 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:05,608 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:05,608 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:05,610 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,610 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,612 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,612 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:05,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cf4f213735744522b2cfb2b88d9e2b68, entries=100, sequenceid=581, filesize=9.5 K 2023-04-03 02:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/aa4300019a7a426c818742635b1b51d0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/aa4300019a7a426c818742635b1b51d0 2023-04-03 02:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47eaf36b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:05,617 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:05,618 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,620 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:05,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/aa4300019a7a426c818742635b1b51d0, entries=100, sequenceid=581, filesize=9.5 K 2023-04-03 02:55:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:05,621 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,621 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:05,621 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for ca59c59c9ebb324c8ea1a6deda6d6369 in 132ms, sequenceid=581, compaction requested=true 2023-04-03 02:55:05,621 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,621 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:05,621 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:05,621 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:55:05,621 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:05,621 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:05,622 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,622 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:05,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:05,623 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37882 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:05,623 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:55:05,623 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,623 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/99fcf2a656bf463db3780f2549e5e89b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/22f85985e1d6421a91393d8eacda66bc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8a2f0dff6eb24ff4a61887e7dd38aad6] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=37.0 K 2023-04-03 02:55:05,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,624 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 99fcf2a656bf463db3780f2549e5e89b, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1680490505190 2023-04-03 02:55:05,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,624 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59712 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:55:05,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,624 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 22f85985e1d6421a91393d8eacda66bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680490505247 2023-04-03 02:55:05,624 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:55:05,625 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,625 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/bb373193ab834c828792b0d9b1f02eec, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cac489b1dd7d4eff96382dab41f31e24, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4b66b8305991458794251db6eefb1b8d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/f80f5b86c6a04496b6635113af3c57d8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cf4f213735744522b2cfb2b88d9e2b68] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=58.3 K 2023-04-03 02:55:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,625 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 8a2f0dff6eb24ff4a61887e7dd38aad6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680490505381 2023-04-03 02:55:05,625 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting bb373193ab834c828792b0d9b1f02eec, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680490504728 2023-04-03 02:55:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,626 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting cac489b1dd7d4eff96382dab41f31e24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1680490504865 2023-04-03 02:55:05,626 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 4b66b8305991458794251db6eefb1b8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1680490505190 2023-04-03 02:55:05,627 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f80f5b86c6a04496b6635113af3c57d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680490505247 2023-04-03 02:55:05,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:05,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:05,627 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:05,627 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:05,627 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:05,627 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:05,627 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:05,627 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting cf4f213735744522b2cfb2b88d9e2b68, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680490505383 2023-04-03 02:55:05,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:05,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:05,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:05,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,632 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,634 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#314 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:05,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,637 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,637 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,637 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,638 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:05,638 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:05,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:05,638 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:05,638 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:05,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:05,638 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,638 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,638 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:05,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:05,638 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:05,638 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:05,641 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#315 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:05,659 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/186cd8c9f83c4b858f9e31051387dfec as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/186cd8c9f83c4b858f9e31051387dfec 2023-04-03 02:55:05,663 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into 186cd8c9f83c4b858f9e31051387dfec(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:05,663 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,663 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=13, startTime=1680490505621; duration=0sec 2023-04-03 02:55:05,663 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:05,663 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:05,664 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35442 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:05,664 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:55:05,664 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,664 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/601265f85f424658b8b66f1757c42875, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/52abd4dfed1a4c36add13d700268a46d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/aa4300019a7a426c818742635b1b51d0] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=34.6 K 2023-04-03 02:55:05,664 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 601265f85f424658b8b66f1757c42875, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1680490505190 2023-04-03 02:55:05,665 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 52abd4dfed1a4c36add13d700268a46d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680490505247 2023-04-03 02:55:05,665 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting aa4300019a7a426c818742635b1b51d0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680490505383 2023-04-03 02:55:05,670 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#316 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:05,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:05,673 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-03 02:55:05,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:05,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:05,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:05,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:05,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:05,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:05,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490565681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490565682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,693 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/6dfd29ebd0ac4c31b820f281bbc35437 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/6dfd29ebd0ac4c31b820f281bbc35437 2023-04-03 02:55:05,697 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into 6dfd29ebd0ac4c31b820f281bbc35437(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:05,697 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:05,697 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=13, startTime=1680490505621; duration=0sec 2023-04-03 02:55:05,697 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:05,739 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:05,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:05,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:05,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:05,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:05,741 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:05,741 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,741 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:05,742 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:05,742 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,742 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,744 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:05,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:05,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:05,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:05,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:05,746 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,746 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:05,746 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:05,746 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:05,748 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,748 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,748 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,748 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:05,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78a9eda5[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:05,750 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:05,750 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,751 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,751 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:05,751 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:05,751 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:05,751 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,752 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:05,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,753 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:05,753 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:05,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,753 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:05,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:05,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:05,755 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:05,755 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:05,755 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:05,755 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:05,755 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:05,758 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,758 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,758 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,758 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,760 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,762 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:05,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:05,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:05,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:05,763 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:05,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,763 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:05,763 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:05,763 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:05,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:05,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490565786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490565786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,863 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:05,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:05,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:05,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:05,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:05,866 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:05,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:05,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:05,867 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,867 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,868 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,868 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:05,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:05,868 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,868 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:05,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:05,869 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:05,869 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:05,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:05,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:05,870 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,870 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:05,870 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:05,870 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:05,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:05,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:05,870 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:05,875 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,875 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,876 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,876 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:05,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3cf813[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:05,878 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:05,878 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,879 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:05,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:05,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:05,879 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,879 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:05,881 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:05,881 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:05,881 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,881 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:05,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:05,884 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:05,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:05,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:05,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:05,884 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:05,886 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,886 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,886 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:05,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:05,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,887 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,890 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:05,890 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:05,890 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:05,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:05,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:05,890 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:05,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:05,891 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:05,890 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,891 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:05,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:05,891 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,891 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,891 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,891 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:05,891 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490565989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:05,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490565989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,991 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:05,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:05,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:05,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:05,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:05,994 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:05,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:05,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:05,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,995 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,996 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,996 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:05,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:05,996 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,996 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:05,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:05,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:05,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:05,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:05,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:05,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:05,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:05,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:05,998 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,998 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:05,998 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:05,998 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:05,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:05,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:05,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:05,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:05,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:05,999 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:06,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:06,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:06,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39a4adf1[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:06,001 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:06,001 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,002 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,003 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:06,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:06,003 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:06,003 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:06,005 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,005 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:06,005 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:06,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:06,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:06,005 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:06,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:06,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:06,007 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:06,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:06,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:06,008 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:06,008 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:06,009 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,009 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:06,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,009 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:06,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:06,010 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:06,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:06,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:06,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,013 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:06,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:06,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:06,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:06,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:06,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,014 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:06,014 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:06,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:06,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:06,068 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/2f5616f3ae2343049f576b0fd8ec768c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/2f5616f3ae2343049f576b0fd8ec768c 2023-04-03 02:55:06,072 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into 2f5616f3ae2343049f576b0fd8ec768c(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:06,073 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:06,073 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=11, startTime=1680490505621; duration=0sec 2023-04-03 02:55:06,073 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:06,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/6db1d3bb42264666b63d73db04b96029 2023-04-03 02:55:06,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/99eea9ecebd04fe5b3b263345a352da6 2023-04-03 02:55:06,114 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:06,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:06,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:06,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:06,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:06,122 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:06,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:06,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:06,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:06,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:06,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:06,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:06,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:06,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:06,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:06,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:06,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/72ca0a8b316349ca9c2bf4d01b7d1c87 2023-04-03 02:55:06,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:06,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:06,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:06,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:06,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:06,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:06,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:06,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:06,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:06,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:06,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:06,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:06,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:06,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:06,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:06,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:06,130 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/6db1d3bb42264666b63d73db04b96029 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6db1d3bb42264666b63d73db04b96029 2023-04-03 02:55:06,131 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:06,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:06,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:06,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:06,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:06,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:06,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:06,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29457005[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:06,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:06,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,135 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6db1d3bb42264666b63d73db04b96029, entries=200, sequenceid=617, filesize=14.3 K 2023-04-03 02:55:06,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:06,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:06,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:06,136 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,136 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:06,136 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/99eea9ecebd04fe5b3b263345a352da6 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/99eea9ecebd04fe5b3b263345a352da6 2023-04-03 02:55:06,137 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:06,137 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:06,137 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:06,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:06,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:06,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:06,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:06,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:06,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/99eea9ecebd04fe5b3b263345a352da6, entries=150, sequenceid=617, filesize=11.9 K 2023-04-03 02:55:06,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/72ca0a8b316349ca9c2bf4d01b7d1c87 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/72ca0a8b316349ca9c2bf4d01b7d1c87 2023-04-03 02:55:06,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:06,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:06,141 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:06,141 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:06,141 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:06,141 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:06,141 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:06,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:06,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:06,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:06,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,146 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/72ca0a8b316349ca9c2bf4d01b7d1c87, entries=150, sequenceid=617, filesize=11.9 K 2023-04-03 02:55:06,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:06,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:06,146 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for ca59c59c9ebb324c8ea1a6deda6d6369 in 473ms, sequenceid=617, compaction requested=false 2023-04-03 02:55:06,147 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,149 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:06,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:06,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:06,149 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:06,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:06,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:06,149 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:06,149 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,149 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:06,149 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:06,149 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:06,149 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,149 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,149 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:06,149 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:06,149 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:06,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:06,250 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:06,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:06,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:06,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:06,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:06,255 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:06,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:06,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:06,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:06,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:06,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,257 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:06,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:06,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:06,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:06,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:06,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:06,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:06,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:06,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:06,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:06,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:06,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:06,260 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:06,260 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:06,260 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 02:55:06,260 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:06,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:06,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:06,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:06,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:06,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:06,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:06,295 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:06,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:06,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490566322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490506311, max timestamp=1680490506324Num uniques 859; 2023-04-03 02:55:06,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:06,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490506311, max timestamp=1680490506324Num uniques 859; 2023-04-03 02:55:06,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:06,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490506311, max timestamp=1680490506324Num uniques 859; 2023-04-03 02:55:06,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:06,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:06,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490566363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:06,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490566426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:06,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490566466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:06,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490566631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,670 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/1e979af78ae24fe287255a1b4fcfe808 2023-04-03 02:55:06,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:06,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490566669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,679 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/e9a6e45dfec0453ba9561777f16fc1fa 2023-04-03 02:55:06,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:06,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490566935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:06,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:06,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490566973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,088 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f8e058c1af63442d810748f04aef5938 2023-04-03 02:55:07,092 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/1e979af78ae24fe287255a1b4fcfe808 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/1e979af78ae24fe287255a1b4fcfe808 2023-04-03 02:55:07,096 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/1e979af78ae24fe287255a1b4fcfe808, entries=150, sequenceid=630, filesize=11.9 K 2023-04-03 02:55:07,097 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/e9a6e45dfec0453ba9561777f16fc1fa as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/e9a6e45dfec0453ba9561777f16fc1fa 2023-04-03 02:55:07,102 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/e9a6e45dfec0453ba9561777f16fc1fa, entries=150, sequenceid=630, filesize=11.9 K 2023-04-03 02:55:07,103 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f8e058c1af63442d810748f04aef5938 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f8e058c1af63442d810748f04aef5938 2023-04-03 02:55:07,106 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f8e058c1af63442d810748f04aef5938, entries=150, sequenceid=630, filesize=11.9 K 2023-04-03 02:55:07,107 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ca59c59c9ebb324c8ea1a6deda6d6369 in 847ms, sequenceid=630, compaction requested=true 2023-04-03 02:55:07,107 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:07,107 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:07,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:07,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:07,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:07,109 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,109 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,109 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,110 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:07,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb6644d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:07,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:07,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,112 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,112 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,113 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:07,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:07,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:07,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:07,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:07,118 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:07,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:07,122 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:07,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:07,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:07,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:07,122 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:07,124 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,124 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,124 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,128 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,128 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:07,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:07,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,129 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:07,129 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:07,229 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:07,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:07,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:07,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:07,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:07,233 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:07,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:07,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:07,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,234 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,234 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,235 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,235 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:07,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:07,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:07,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:07,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:07,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:07,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:07,236 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,236 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:07,237 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-03 02:55:07,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:07,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:07,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:07,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:07,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:07,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:07,246 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/4ad361424b3544f2bdf7fc6927f78649 2023-04-03 02:55:07,255 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/cc16b7829ea347f4b2363fc9df3e8fb6 2023-04-03 02:55:07,268 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/311eb4b61d5b42f2bd38d4fc26754dbd 2023-04-03 02:55:07,271 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/4ad361424b3544f2bdf7fc6927f78649 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/4ad361424b3544f2bdf7fc6927f78649 2023-04-03 02:55:07,275 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/4ad361424b3544f2bdf7fc6927f78649, entries=150, sequenceid=665, filesize=11.9 K 2023-04-03 02:55:07,275 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/cc16b7829ea347f4b2363fc9df3e8fb6 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cc16b7829ea347f4b2363fc9df3e8fb6 2023-04-03 02:55:07,278 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cc16b7829ea347f4b2363fc9df3e8fb6, entries=150, sequenceid=665, filesize=11.9 K 2023-04-03 02:55:07,279 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/311eb4b61d5b42f2bd38d4fc26754dbd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/311eb4b61d5b42f2bd38d4fc26754dbd 2023-04-03 02:55:07,282 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/311eb4b61d5b42f2bd38d4fc26754dbd, entries=150, sequenceid=665, filesize=11.9 K 2023-04-03 02:55:07,283 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for ca59c59c9ebb324c8ea1a6deda6d6369 in 47ms, sequenceid=665, compaction requested=true 2023-04-03 02:55:07,283 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:07,283 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:07,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,285 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,286 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,286 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:07,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3561ebe8[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:07,287 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:07,288 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,289 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,289 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:07,289 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:07,289 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:07,289 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,289 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:07,290 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:07,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:07,293 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:07,293 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:07,293 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:07,293 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:07,293 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:07,295 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,295 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,295 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,296 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,298 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,301 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:07,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:07,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,302 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:07,302 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:07,402 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:07,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:07,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:07,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:07,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:07,405 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:07,405 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,405 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:07,405 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:07,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,407 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,407 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,407 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,407 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:07,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:07,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:07,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:07,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:07,408 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:07,408 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,408 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:07,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:07,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:07,409 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,409 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:07,409 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:07,409 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:07,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:07,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:07,409 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:07,411 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,411 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,411 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,411 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:07,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29b61390[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:07,413 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:07,413 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,414 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,414 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:07,414 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:07,414 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:07,414 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,414 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:07,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:07,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,416 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:07,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:07,419 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:07,419 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:07,419 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:07,419 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:07,419 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:07,421 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,421 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,421 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,421 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,423 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:07,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:07,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,426 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:07,426 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:07,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,426 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,425 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,426 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:07,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:07,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:55:07,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:07,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:07,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:07,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:07,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:07,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:07,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490507496, max timestamp=1680490507504Num uniques 859; 2023-04-03 02:55:07,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490507496, max timestamp=1680490507504Num uniques 859; 2023-04-03 02:55:07,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:07,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:07,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490507496, max timestamp=1680490507504Num uniques 859; 2023-04-03 02:55:07,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:07,526 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:07,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:07,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:07,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:07,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:07,529 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:07,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:07,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:07,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:07,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:07,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:07,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:07,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:07,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:07,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:07,532 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,532 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:07,532 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:07,533 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:07,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:07,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:07,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:07,535 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,535 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,536 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,536 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bf19bfb[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:07,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:07,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,540 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:07,540 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:07,540 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:07,540 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,540 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:07,541 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,541 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:07,541 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,541 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:07,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:07,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490567539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:07,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490567540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:07,544 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:07,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:07,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:07,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:07,545 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:07,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,557 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,560 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,560 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,560 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,560 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:07,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:07,561 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:07,561 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:07,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:07,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490567645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:07,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490567646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,661 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:07,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:07,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:07,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:07,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:07,669 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:07,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:07,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:07,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,671 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:07,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:07,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:07,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:07,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:07,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:07,688 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,688 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:07,688 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:07,688 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:07,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:07,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:07,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,691 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:07,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64806c4[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:07,694 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:07,694 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,695 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:07,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:07,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:07,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:07,697 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:07,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:07,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:07,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:07,700 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:07,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:07,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:07,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:07,700 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:07,702 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,702 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,702 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,703 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,709 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,713 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:07,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:07,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,713 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:07,713 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:07,813 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:07,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:07,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:07,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:07,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:07,818 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:07,818 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,818 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:07,818 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:07,818 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,819 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,820 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,820 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,820 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,820 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:07,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:07,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:07,822 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,822 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:07,822 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:07,822 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:07,824 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,824 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,824 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,824 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:07,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ccc934e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:07,826 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:07,826 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,827 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:07,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:07,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:07,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:07,829 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:07,831 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:07,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:07,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:07,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:07,832 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:07,833 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,833 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,833 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,835 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:07,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:07,838 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:07,838 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,838 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,838 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,838 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,838 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:07,838 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:07,838 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:07,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490567851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:07,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490567851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=676 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/b642cf1d826a4d9da5dddbfe7ef0a2c0 2023-04-03 02:55:07,938 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:07,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:07,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:07,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:07,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:07,942 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:07,942 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,942 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:07,942 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:07,942 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,942 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,943 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,944 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,944 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,944 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:07,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:07,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:07,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:07,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:07,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:07,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:07,945 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,945 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:07,946 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:07,946 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:07,948 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25ff3dd4[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:07,950 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:07,950 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,951 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,951 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:07,951 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:07,951 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,951 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:07,951 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:07,952 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,952 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:07,953 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,953 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:07,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:07,955 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:07,955 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:07,955 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:07,955 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:07,955 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:07,956 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:07,956 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,956 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:07,957 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,958 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:07,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,964 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,964 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,964 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:07,964 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:07,964 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:07,964 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:07,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:07,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:07,964 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:07,964 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:07,964 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:07,965 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:07,964 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:07,965 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:07,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:08,065 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:08,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:08,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:08,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:08,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:08,068 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:08,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:08,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:08,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,069 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:08,069 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:08,070 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,070 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:08,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:08,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:08,071 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,071 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:08,071 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:08,071 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:08,071 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:08,073 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,073 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,073 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,073 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:08,075 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:08,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27644d3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:08,075 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,077 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:08,077 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,077 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:08,078 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:08,078 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:08,078 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,078 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:08,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:08,080 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:08,080 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:08,080 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:08,080 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:08,080 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:08,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:08,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:08,082 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,083 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,085 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,085 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,085 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,085 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,085 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:08,085 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:08,085 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:08,085 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,085 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,085 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:08,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:08,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:08,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:08,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:08,086 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:08,086 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:08,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:08,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490568158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:08,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490568158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,186 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:08,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:08,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:08,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:08,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:08,190 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:08,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:08,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:08,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,194 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:08,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:08,194 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,195 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,195 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:08,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:08,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:08,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:08,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:08,196 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,196 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:08,196 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:08,196 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:08,197 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,197 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:08,197 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:08,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,199 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,199 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,199 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:08,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@444e1c7f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:08,201 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:08,201 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,203 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:08,203 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,203 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:08,205 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:08,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:08,205 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:08,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:08,216 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:08,216 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:08,216 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:08,216 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:08,216 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:08,219 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,219 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,219 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:08,220 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,222 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,225 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,225 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:08,225 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,225 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:08,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:08,225 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,226 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,226 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,226 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,226 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,226 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:08,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:08,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:08,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:08,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:08,226 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:08,226 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:08,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=676 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/c40ec9439b6c4d50bc1633d9540c007b 2023-04-03 02:55:08,326 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:08,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:08,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:08,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:08,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:08,329 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:08,329 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,329 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:08,329 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:08,330 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,330 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,331 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:08,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:08,331 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,331 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:08,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:08,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:08,332 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:08,332 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,332 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:08,333 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,333 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:08,333 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:08,333 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:08,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:08,334 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:08,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cc0a15b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:08,336 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:08,336 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,338 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,338 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:08,338 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:08,338 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,338 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:08,338 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:08,340 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,340 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:08,340 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,340 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:08,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:08,342 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:08,342 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:08,342 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:08,342 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:08,342 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:08,343 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,343 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,343 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:08,344 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,345 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:08,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:08,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:08,348 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:08,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:08,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:08,348 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,348 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:08,348 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,348 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,348 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,348 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:08,348 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:08,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:08,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:08,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33756 deadline: 1680490568440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,444 DEBUG [Thread-1532] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:08,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33770 deadline: 1680490568447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,448 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:08,449 DEBUG [Thread-1526] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:08,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:08,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:08,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:08,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33750 deadline: 1680490568459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:08,462 DEBUG [Thread-1524] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:08,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:08,462 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:08,463 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,463 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:08,463 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:08,463 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,463 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,467 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,467 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:08,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:08,468 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,468 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:08,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:08,469 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,469 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:08,469 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:08,469 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:08,471 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,471 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,471 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,471 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:08,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408d3e8a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:08,473 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:08,473 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,473 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:08,474 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,474 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:08,476 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,476 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:08,476 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,476 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:08,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:08,478 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:08,478 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:08,478 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:08,478 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:08,478 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:08,480 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,480 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,480 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:08,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,481 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,482 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,484 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,484 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:08,484 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:08,484 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:08,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:08,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:08,484 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:08,484 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,484 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,484 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,484 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,484 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,484 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:08,484 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:08,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:08,584 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:08,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:08,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:08,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:08,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:08,588 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:08,588 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,588 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:08,588 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:08,588 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,588 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,589 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:08,589 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:08,590 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,590 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:08,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:08,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:08,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:08,591 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,591 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:08,591 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:08,591 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:08,591 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:08,599 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,599 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,599 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,599 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:08,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:08,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@706daa94[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:08,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,602 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,602 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:08,602 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:08,602 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:08,602 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,602 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:08,603 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,603 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:08,603 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,603 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:08,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:08,606 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:08,606 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:08,606 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:08,606 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:08,606 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:08,607 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:08,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:08,607 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,608 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:08,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:08,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,609 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:08,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:08,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,612 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,612 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,612 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,612 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,612 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,612 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:08,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:08,612 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:08,612 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:08,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:08,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:08,612 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:08,612 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:08,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:08,612 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:08,612 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:08,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:08,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:08,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:08,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33692 deadline: 1680490568660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:33720 deadline: 1680490568660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=676 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/559edb8a4b1b49ed9bc0e4c3da2152fc 2023-04-03 02:55:08,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/b642cf1d826a4d9da5dddbfe7ef0a2c0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b642cf1d826a4d9da5dddbfe7ef0a2c0 2023-04-03 02:55:08,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b642cf1d826a4d9da5dddbfe7ef0a2c0, entries=200, sequenceid=676, filesize=14.3 K 2023-04-03 02:55:08,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/c40ec9439b6c4d50bc1633d9540c007b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c40ec9439b6c4d50bc1633d9540c007b 2023-04-03 02:55:08,708 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c40ec9439b6c4d50bc1633d9540c007b, entries=150, sequenceid=676, filesize=11.9 K 2023-04-03 02:55:08,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/559edb8a4b1b49ed9bc0e4c3da2152fc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/559edb8a4b1b49ed9bc0e4c3da2152fc 2023-04-03 02:55:08,713 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:08,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:08,713 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/559edb8a4b1b49ed9bc0e4c3da2152fc, entries=150, sequenceid=676, filesize=11.9 K 2023-04-03 02:55:08,714 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ca59c59c9ebb324c8ea1a6deda6d6369 in 1263ms, sequenceid=676, compaction requested=true 2023-04-03 02:55:08,714 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:08,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:08,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:08,715 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:55:08,715 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:55:08,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:55:08,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:08,717 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 67240 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:55:08,717 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62326 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:55:08,718 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/A is initiating minor compaction (all files) 2023-04-03 02:55:08,718 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/B is initiating minor compaction (all files) 2023-04-03 02:55:08,718 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/A in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,718 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/B in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,718 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/186cd8c9f83c4b858f9e31051387dfec, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6db1d3bb42264666b63d73db04b96029, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/1e979af78ae24fe287255a1b4fcfe808, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/4ad361424b3544f2bdf7fc6927f78649, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b642cf1d826a4d9da5dddbfe7ef0a2c0] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=65.7 K 2023-04-03 02:55:08,718 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/2f5616f3ae2343049f576b0fd8ec768c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/99eea9ecebd04fe5b3b263345a352da6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/e9a6e45dfec0453ba9561777f16fc1fa, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cc16b7829ea347f4b2363fc9df3e8fb6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c40ec9439b6c4d50bc1633d9540c007b] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=60.9 K 2023-04-03 02:55:08,718 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 2f5616f3ae2343049f576b0fd8ec768c, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680490505252 2023-04-03 02:55:08,719 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 99eea9ecebd04fe5b3b263345a352da6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1680490505558 2023-04-03 02:55:08,719 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e9a6e45dfec0453ba9561777f16fc1fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1680490505675 2023-04-03 02:55:08,720 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting cc16b7829ea347f4b2363fc9df3e8fb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680490506355 2023-04-03 02:55:08,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:08,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:08,720 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:08,720 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,720 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c40ec9439b6c4d50bc1633d9540c007b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=676, earliestPutTs=1680490507446 2023-04-03 02:55:08,720 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:08,720 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:08,720 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,720 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,721 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:08,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:08,722 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:08,722 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,722 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 186cd8c9f83c4b858f9e31051387dfec, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680490505252 2023-04-03 02:55:08,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:08,723 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 6db1d3bb42264666b63d73db04b96029, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1680490505558 2023-04-03 02:55:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:08,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:08,723 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 1e979af78ae24fe287255a1b4fcfe808, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1680490505675 2023-04-03 02:55:08,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:08,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:08,724 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:08,724 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:08,724 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:08,724 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 4ad361424b3544f2bdf7fc6927f78649, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680490506355 2023-04-03 02:55:08,724 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b642cf1d826a4d9da5dddbfe7ef0a2c0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=676, earliestPutTs=1680490507444 2023-04-03 02:55:08,728 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:08,728 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:08,728 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:08,728 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. started... 2023-04-03 02:55:08,728 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-03 02:55:08,728 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:08,728 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:08,728 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:08,728 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:08,729 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:08,729 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:08,731 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#B#compaction#329 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:08,742 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#A#compaction#331 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:09,159 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/0e217be2ad274049846d594679776950 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0e217be2ad274049846d594679776950 2023-04-03 02:55:09,163 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/B of ca59c59c9ebb324c8ea1a6deda6d6369 into 0e217be2ad274049846d594679776950(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:09,164 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:09,164 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/B, priority=11, startTime=1680490508715; duration=0sec 2023-04-03 02:55:09,164 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:09,164 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:55:09,165 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/cde4628cf7564791b5d033ba01f9e144 2023-04-03 02:55:09,165 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62360 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:55:09,166 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): ca59c59c9ebb324c8ea1a6deda6d6369/C is initiating minor compaction (all files) 2023-04-03 02:55:09,166 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca59c59c9ebb324c8ea1a6deda6d6369/C in TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:09,166 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/6dfd29ebd0ac4c31b820f281bbc35437, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/72ca0a8b316349ca9c2bf4d01b7d1c87, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f8e058c1af63442d810748f04aef5938, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/311eb4b61d5b42f2bd38d4fc26754dbd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/559edb8a4b1b49ed9bc0e4c3da2152fc] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp, totalSize=60.9 K 2023-04-03 02:55:09,166 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6dfd29ebd0ac4c31b820f281bbc35437, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680490505252 2023-04-03 02:55:09,167 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 72ca0a8b316349ca9c2bf4d01b7d1c87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1680490505558 2023-04-03 02:55:09,167 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting f8e058c1af63442d810748f04aef5938, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1680490505675 2023-04-03 02:55:09,167 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 311eb4b61d5b42f2bd38d4fc26754dbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680490506355 2023-04-03 02:55:09,168 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 559edb8a4b1b49ed9bc0e4c3da2152fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=676, earliestPutTs=1680490507446 2023-04-03 02:55:09,171 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/b7fd6a886c544ce3b339ab394116126c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b7fd6a886c544ce3b339ab394116126c 2023-04-03 02:55:09,177 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/A of ca59c59c9ebb324c8ea1a6deda6d6369 into b7fd6a886c544ce3b339ab394116126c(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:09,177 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:09,177 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/A, priority=11, startTime=1680490508714; duration=0sec 2023-04-03 02:55:09,177 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:09,180 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca59c59c9ebb324c8ea1a6deda6d6369#C#compaction#333 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:09,197 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/f90dde3b400a45d4afd431e5cc1e59c2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f90dde3b400a45d4afd431e5cc1e59c2 2023-04-03 02:55:09,201 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ca59c59c9ebb324c8ea1a6deda6d6369/C of ca59c59c9ebb324c8ea1a6deda6d6369 into f90dde3b400a45d4afd431e5cc1e59c2(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:09,202 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:09,202 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369., storeName=ca59c59c9ebb324c8ea1a6deda6d6369/C, priority=11, startTime=1680490508715; duration=0sec 2023-04-03 02:55:09,202 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:09,356 DEBUG [Thread-1537] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:52506 2023-04-03 02:55:09,356 DEBUG [Thread-1537] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:09,357 DEBUG [Thread-1539] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:52506 2023-04-03 02:55:09,357 DEBUG [Thread-1535] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x753156b2 to 127.0.0.1:52506 2023-04-03 02:55:09,357 DEBUG [Thread-1535] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:09,357 DEBUG [Thread-1539] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:09,358 DEBUG [Thread-1541] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:52506 2023-04-03 02:55:09,358 DEBUG [Thread-1541] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:09,359 DEBUG [Thread-1543] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de064cb to 127.0.0.1:52506 2023-04-03 02:55:09,359 DEBUG [Thread-1543] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:09,585 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/6f7d5fed1620412daac1f4df69465267 2023-04-03 02:55:09,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:09,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. as already flushing 2023-04-03 02:55:09,667 DEBUG [Thread-1528] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:52506 2023-04-03 02:55:09,667 DEBUG [Thread-1528] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:09,671 DEBUG [Thread-1530] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:52506 2023-04-03 02:55:09,671 DEBUG [Thread-1530] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:09,994 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/8515d105ddc644b8bb8fa4150d0a4a9e 2023-04-03 02:55:09,997 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/cde4628cf7564791b5d033ba01f9e144 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cde4628cf7564791b5d033ba01f9e144 2023-04-03 02:55:09,999 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cde4628cf7564791b5d033ba01f9e144, entries=150, sequenceid=712, filesize=11.9 K 2023-04-03 02:55:10,000 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/6f7d5fed1620412daac1f4df69465267 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6f7d5fed1620412daac1f4df69465267 2023-04-03 02:55:10,003 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6f7d5fed1620412daac1f4df69465267, entries=150, sequenceid=712, filesize=11.9 K 2023-04-03 02:55:10,003 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/8515d105ddc644b8bb8fa4150d0a4a9e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8515d105ddc644b8bb8fa4150d0a4a9e 2023-04-03 02:55:10,006 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8515d105ddc644b8bb8fa4150d0a4a9e, entries=150, sequenceid=712, filesize=11.9 K 2023-04-03 02:55:10,007 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=13.42 KB/13740 for ca59c59c9ebb324c8ea1a6deda6d6369 in 1279ms, sequenceid=712, compaction requested=false 2023-04-03 02:55:10,007 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:10,007 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:10,007 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:10,007 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:10,007 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,007 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:10,007 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:10,010 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:10,010 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:10,010 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:10,010 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f4ed02[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:10,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:10,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:10,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:10,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:10,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:10,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:10,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:10,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:10,013 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,013 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:10,014 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:10,014 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:10,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,014 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:10,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:10,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:10,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:10,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:10,016 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:10,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:10,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:10,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:10,016 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:10,017 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:10,017 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:10,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:10,018 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:10,018 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:10,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:10,019 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:10,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:10,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:10,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:10,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:10,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:10,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:10,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:10,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:10,024 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:10,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:10,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:10,024 DEBUG [Thread-1534] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:10,024 DEBUG [Thread-1534] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:10,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:10,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:10,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:10,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:10,024 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:10,124 DEBUG [Thread-1534] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:10,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:18,465 DEBUG [Thread-1532] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7810a231 to 127.0.0.1:52506 2023-04-03 02:55:18,465 DEBUG [Thread-1532] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:18,505 DEBUG [Thread-1526] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:52506 2023-04-03 02:55:18,505 DEBUG [Thread-1526] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:18,529 DEBUG [Thread-1524] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:52506 2023-04-03 02:55:18,529 DEBUG [Thread-1524] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:18,529 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 02:55:18,529 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 56 2023-04-03 02:55:18,529 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 62 2023-04-03 02:55:18,529 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 190 2023-04-03 02:55:18,530 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 196 2023-04-03 02:55:18,530 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 68 2023-04-03 02:55:18,530 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 02:55:18,530 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 5100 2023-04-03 02:55:18,530 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 4943 2023-04-03 02:55:18,530 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 4993 2023-04-03 02:55:18,530 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 5100 2023-04-03 02:55:18,530 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 4925 2023-04-03 02:55:18,530 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 02:55:18,530 INFO [Listener at localhost/46853] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 02:55:18,530 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ed1dde9 to 127.0.0.1:52506 2023-04-03 02:55:18,530 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:18,530 INFO [Listener at localhost/46853] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 02:55:18,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 02:55:18,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:18,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 02:55:18,534 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490518534"}]},"ts":"1680490518534"} 2023-04-03 02:55:18,535 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 02:55:18,536 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 02:55:18,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca59c59c9ebb324c8ea1a6deda6d6369, UNASSIGN}] 2023-04-03 02:55:18,538 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca59c59c9ebb324c8ea1a6deda6d6369, UNASSIGN 2023-04-03 02:55:18,538 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=ca59c59c9ebb324c8ea1a6deda6d6369, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:18,538 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490518538"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490518538"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490518538"}]},"ts":"1680490518538"} 2023-04-03 02:55:18,539 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:55:18,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 02:55:18,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:18,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ca59c59c9ebb324c8ea1a6deda6d6369, disabling compactions & flushes 2023-04-03 02:55:18,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:18,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:18,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. after waiting 0 ms 2023-04-03 02:55:18,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:18,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing ca59c59c9ebb324c8ea1a6deda6d6369 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-03 02:55:18,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=A 2023-04-03 02:55:18,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:18,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=B 2023-04-03 02:55:18,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:18,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ca59c59c9ebb324c8ea1a6deda6d6369, store=C 2023-04-03 02:55:18,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:18,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 02:55:19,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/42392e45efa549c6bf9eff7267673ac2 2023-04-03 02:55:19,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 02:55:19,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/3ab3a0e4603a46369866976d72aad31d 2023-04-03 02:55:19,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 02:55:19,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/1ca3718667f045ceb170e54b52425fbf 2023-04-03 02:55:19,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/A/42392e45efa549c6bf9eff7267673ac2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/42392e45efa549c6bf9eff7267673ac2 2023-04-03 02:55:19,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/42392e45efa549c6bf9eff7267673ac2, entries=150, sequenceid=723, filesize=11.9 K 2023-04-03 02:55:19,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/B/3ab3a0e4603a46369866976d72aad31d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/3ab3a0e4603a46369866976d72aad31d 2023-04-03 02:55:19,926 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/3ab3a0e4603a46369866976d72aad31d, entries=150, sequenceid=723, filesize=11.9 K 2023-04-03 02:55:19,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/.tmp/C/1ca3718667f045ceb170e54b52425fbf as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1ca3718667f045ceb170e54b52425fbf 2023-04-03 02:55:19,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1ca3718667f045ceb170e54b52425fbf, entries=150, sequenceid=723, filesize=11.9 K 2023-04-03 02:55:19,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for ca59c59c9ebb324c8ea1a6deda6d6369 in 1239ms, sequenceid=723, compaction requested=true 2023-04-03 02:55:19,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2c286d20d6c24bcfa7da8a88e4f1876b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c176ae78cb1f4fe0bc45fa78dfd76a67, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/09c9e5a08e214ecf87edddb5174e529d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/00b68aa8332d413e8c25dbf7d508a546, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/016780c8a84d4b2fa6a337a66d0669a9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8389be08afdf450aa8c495eab0d6051d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/dc96198d5d3f4b31801f402de5255d18, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60c06a7dbe2d45f2b37d35cb58d46278, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/1a46f3358d514955ad19ebea007efba7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/99245e0a51ee4b4480b43449161285d1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c618691566884fee96a3a0d2cf4c4811, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c691d2f954834682b9bca5320a181816, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0dd724da41a14388811b91332dc88b95, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0e63d68321414df18a947c75a9b1735f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/4a853b1b4d134d6d8af38c18092cba82, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/a75cf1f3c27d467083ec43cb07365ea6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8b537e002f2d4e9aa5f61253ea721ad1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/13eccae1feac4c49b75818433f4c1be9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cff0733d39844d33a5e17a0e7ec3adbd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9d7365adccca4b7aab32121457545bbc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c714ec82d888461d8d7d28bcf4d59156, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2f941fffe7fb46188dac6c2bc393255e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/088aa43e292a4460834a3dc5504df986, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eaf1dd9c7f1f4b93ac92f163a42371af, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6d6aec2731474295828d31c5a683d1b0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/53b9bbb3a4e14ea78524126fbaf37dbe, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9e08bb9f6eba4fbcbbc11eaa37d150a9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eb2e0d6d66a8423f8d29695ae7d5c578, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/5f3112bc838143c69b4e859eff0e7131, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/700f20793c534a9fa9861e64ca6a0936, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60edeec55c264fafbdeeae74bc47851b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eac2135b06d241c4b2130f64f55fc802, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b10a4a61eb904c72b341a97a814f9cf3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/06643d40c4d54cb1b6733cdc16b3abb4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/dce52598bd5c4af39b72ac7b4a73376e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/562dbddbbef648f0b0e00eaff93dcdb2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cd0d8fc76f0b44309912ae6bab2cc0ed, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0a32b2ef97c64959b4dec90b3ebb1394, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/ae993ab9052b466db9fe79e55eff5fc8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/99fcf2a656bf463db3780f2549e5e89b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/22f85985e1d6421a91393d8eacda66bc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/186cd8c9f83c4b858f9e31051387dfec, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8a2f0dff6eb24ff4a61887e7dd38aad6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6db1d3bb42264666b63d73db04b96029, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/1e979af78ae24fe287255a1b4fcfe808, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/4ad361424b3544f2bdf7fc6927f78649, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b642cf1d826a4d9da5dddbfe7ef0a2c0] to archive 2023-04-03 02:55:19,954 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:55:19,955 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2c286d20d6c24bcfa7da8a88e4f1876b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2c286d20d6c24bcfa7da8a88e4f1876b 2023-04-03 02:55:19,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c176ae78cb1f4fe0bc45fa78dfd76a67 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c176ae78cb1f4fe0bc45fa78dfd76a67 2023-04-03 02:55:19,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/09c9e5a08e214ecf87edddb5174e529d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/09c9e5a08e214ecf87edddb5174e529d 2023-04-03 02:55:19,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/00b68aa8332d413e8c25dbf7d508a546 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/00b68aa8332d413e8c25dbf7d508a546 2023-04-03 02:55:19,959 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/016780c8a84d4b2fa6a337a66d0669a9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/016780c8a84d4b2fa6a337a66d0669a9 2023-04-03 02:55:19,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8389be08afdf450aa8c495eab0d6051d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8389be08afdf450aa8c495eab0d6051d 2023-04-03 02:55:19,961 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/dc96198d5d3f4b31801f402de5255d18 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/dc96198d5d3f4b31801f402de5255d18 2023-04-03 02:55:19,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60c06a7dbe2d45f2b37d35cb58d46278 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60c06a7dbe2d45f2b37d35cb58d46278 2023-04-03 02:55:19,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/1a46f3358d514955ad19ebea007efba7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/1a46f3358d514955ad19ebea007efba7 2023-04-03 02:55:19,963 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/99245e0a51ee4b4480b43449161285d1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/99245e0a51ee4b4480b43449161285d1 2023-04-03 02:55:19,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c618691566884fee96a3a0d2cf4c4811 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c618691566884fee96a3a0d2cf4c4811 2023-04-03 02:55:19,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c691d2f954834682b9bca5320a181816 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c691d2f954834682b9bca5320a181816 2023-04-03 02:55:19,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0dd724da41a14388811b91332dc88b95 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0dd724da41a14388811b91332dc88b95 2023-04-03 02:55:19,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0e63d68321414df18a947c75a9b1735f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0e63d68321414df18a947c75a9b1735f 2023-04-03 02:55:19,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/4a853b1b4d134d6d8af38c18092cba82 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/4a853b1b4d134d6d8af38c18092cba82 2023-04-03 02:55:19,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/a75cf1f3c27d467083ec43cb07365ea6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/a75cf1f3c27d467083ec43cb07365ea6 2023-04-03 02:55:19,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8b537e002f2d4e9aa5f61253ea721ad1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8b537e002f2d4e9aa5f61253ea721ad1 2023-04-03 02:55:19,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/13eccae1feac4c49b75818433f4c1be9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/13eccae1feac4c49b75818433f4c1be9 2023-04-03 02:55:19,970 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cff0733d39844d33a5e17a0e7ec3adbd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cff0733d39844d33a5e17a0e7ec3adbd 2023-04-03 02:55:19,971 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9d7365adccca4b7aab32121457545bbc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9d7365adccca4b7aab32121457545bbc 2023-04-03 02:55:19,972 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c714ec82d888461d8d7d28bcf4d59156 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/c714ec82d888461d8d7d28bcf4d59156 2023-04-03 02:55:19,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2f941fffe7fb46188dac6c2bc393255e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/2f941fffe7fb46188dac6c2bc393255e 2023-04-03 02:55:19,974 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/088aa43e292a4460834a3dc5504df986 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/088aa43e292a4460834a3dc5504df986 2023-04-03 02:55:19,975 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eaf1dd9c7f1f4b93ac92f163a42371af to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eaf1dd9c7f1f4b93ac92f163a42371af 2023-04-03 02:55:19,975 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6d6aec2731474295828d31c5a683d1b0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6d6aec2731474295828d31c5a683d1b0 2023-04-03 02:55:19,976 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/53b9bbb3a4e14ea78524126fbaf37dbe to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/53b9bbb3a4e14ea78524126fbaf37dbe 2023-04-03 02:55:19,977 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9e08bb9f6eba4fbcbbc11eaa37d150a9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/9e08bb9f6eba4fbcbbc11eaa37d150a9 2023-04-03 02:55:19,978 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eb2e0d6d66a8423f8d29695ae7d5c578 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eb2e0d6d66a8423f8d29695ae7d5c578 2023-04-03 02:55:19,979 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/5f3112bc838143c69b4e859eff0e7131 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/5f3112bc838143c69b4e859eff0e7131 2023-04-03 02:55:19,980 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/700f20793c534a9fa9861e64ca6a0936 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/700f20793c534a9fa9861e64ca6a0936 2023-04-03 02:55:19,981 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60edeec55c264fafbdeeae74bc47851b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/60edeec55c264fafbdeeae74bc47851b 2023-04-03 02:55:19,981 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eac2135b06d241c4b2130f64f55fc802 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/eac2135b06d241c4b2130f64f55fc802 2023-04-03 02:55:19,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b10a4a61eb904c72b341a97a814f9cf3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b10a4a61eb904c72b341a97a814f9cf3 2023-04-03 02:55:19,983 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/06643d40c4d54cb1b6733cdc16b3abb4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/06643d40c4d54cb1b6733cdc16b3abb4 2023-04-03 02:55:19,984 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/dce52598bd5c4af39b72ac7b4a73376e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/dce52598bd5c4af39b72ac7b4a73376e 2023-04-03 02:55:19,985 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/562dbddbbef648f0b0e00eaff93dcdb2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/562dbddbbef648f0b0e00eaff93dcdb2 2023-04-03 02:55:19,986 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cd0d8fc76f0b44309912ae6bab2cc0ed to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cd0d8fc76f0b44309912ae6bab2cc0ed 2023-04-03 02:55:19,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0a32b2ef97c64959b4dec90b3ebb1394 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/0a32b2ef97c64959b4dec90b3ebb1394 2023-04-03 02:55:19,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/ae993ab9052b466db9fe79e55eff5fc8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/ae993ab9052b466db9fe79e55eff5fc8 2023-04-03 02:55:19,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/99fcf2a656bf463db3780f2549e5e89b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/99fcf2a656bf463db3780f2549e5e89b 2023-04-03 02:55:19,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/22f85985e1d6421a91393d8eacda66bc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/22f85985e1d6421a91393d8eacda66bc 2023-04-03 02:55:19,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/186cd8c9f83c4b858f9e31051387dfec to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/186cd8c9f83c4b858f9e31051387dfec 2023-04-03 02:55:19,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8a2f0dff6eb24ff4a61887e7dd38aad6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/8a2f0dff6eb24ff4a61887e7dd38aad6 2023-04-03 02:55:19,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6db1d3bb42264666b63d73db04b96029 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/6db1d3bb42264666b63d73db04b96029 2023-04-03 02:55:19,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/1e979af78ae24fe287255a1b4fcfe808 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/1e979af78ae24fe287255a1b4fcfe808 2023-04-03 02:55:19,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/4ad361424b3544f2bdf7fc6927f78649 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/4ad361424b3544f2bdf7fc6927f78649 2023-04-03 02:55:19,996 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b642cf1d826a4d9da5dddbfe7ef0a2c0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b642cf1d826a4d9da5dddbfe7ef0a2c0 2023-04-03 02:55:20,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d806662321f843bba0120d8e025e62d9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4ebd707b4e41478399da8a1921d66434, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b48e086d32294bf3ba0de888e9f1d469, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7dbbda6e5d6e428f95359d3be9978309, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6d073555a4414f3095c5f93d0664f644, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/494aca889df8480f9fed66526ff19cc4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/050d7d353a054a35b51d2af279b14aa4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c9845fd868944621960d01401c265dbe, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d7a778598d0e4b1e96874ab4766ee681, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/434d0e6cb89f455d960dbbb28b0cda74, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0ec2502211434a7d85e26caa5ec3fd2d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7c5cb48d542f4cb3ab13c046c51b42dc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/07569886019c4db187084410ade7e28e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/61efeaaca5e249ac851c3c8a6f53ecd5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/ab07476b983c46eb8fe02a696068c5bd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/bda2cd5ace9a4336a2b743214a5a9545, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/faee7a1237194568aa63bd94ea82b3bf, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/3629a3bdf3bd4e8882b613c95b62d98e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/5bf493a7c1a34d86a1722cc7d4c1209d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/142a7e82795244a7a6976ccd4482c505, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/73ee2df950c64b9c910d99a5925cd877, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7e70cff08f754ac2a247a78d87d7d199, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/76436b28b8e54fff88f9ef510cb78a10, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b3a42467107f47948d589ecea870418a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d5a384bed07f492abe1d99b1925aaca4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0241ba6d0c6d4d31acce5550754cf4e8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4d4fbf8bb05e4c229708cbf0fe7b35a3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6f841e787f95452dbfac980f1b5e4f4b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/ac40fb1251e94acb8079874d09505110, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c61798973a7c4a70b3684e4f1fe38fdd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b174603456a34e36930b26da989ee291, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/a5b3b76c61b04bb696a372814202584a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/70eefa62ea5d4d71a5808bc9bee77857, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/577b6b46ad4c4042b7466e3aba241654, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/52a188bb271842f2a3ab1d6b1d02d155, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/bb373193ab834c828792b0d9b1f02eec, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7d35d5adc73149f184bc53ba56bb488f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cac489b1dd7d4eff96382dab41f31e24, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4b66b8305991458794251db6eefb1b8d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/f80f5b86c6a04496b6635113af3c57d8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/2f5616f3ae2343049f576b0fd8ec768c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cf4f213735744522b2cfb2b88d9e2b68, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/99eea9ecebd04fe5b3b263345a352da6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/e9a6e45dfec0453ba9561777f16fc1fa, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cc16b7829ea347f4b2363fc9df3e8fb6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c40ec9439b6c4d50bc1633d9540c007b] to archive 2023-04-03 02:55:20,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:55:20,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d806662321f843bba0120d8e025e62d9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d806662321f843bba0120d8e025e62d9 2023-04-03 02:55:20,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4ebd707b4e41478399da8a1921d66434 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4ebd707b4e41478399da8a1921d66434 2023-04-03 02:55:20,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b48e086d32294bf3ba0de888e9f1d469 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b48e086d32294bf3ba0de888e9f1d469 2023-04-03 02:55:20,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7dbbda6e5d6e428f95359d3be9978309 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7dbbda6e5d6e428f95359d3be9978309 2023-04-03 02:55:20,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6d073555a4414f3095c5f93d0664f644 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6d073555a4414f3095c5f93d0664f644 2023-04-03 02:55:20,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/494aca889df8480f9fed66526ff19cc4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/494aca889df8480f9fed66526ff19cc4 2023-04-03 02:55:20,008 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/050d7d353a054a35b51d2af279b14aa4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/050d7d353a054a35b51d2af279b14aa4 2023-04-03 02:55:20,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c9845fd868944621960d01401c265dbe to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c9845fd868944621960d01401c265dbe 2023-04-03 02:55:20,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d7a778598d0e4b1e96874ab4766ee681 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d7a778598d0e4b1e96874ab4766ee681 2023-04-03 02:55:20,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/434d0e6cb89f455d960dbbb28b0cda74 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/434d0e6cb89f455d960dbbb28b0cda74 2023-04-03 02:55:20,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0ec2502211434a7d85e26caa5ec3fd2d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0ec2502211434a7d85e26caa5ec3fd2d 2023-04-03 02:55:20,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7c5cb48d542f4cb3ab13c046c51b42dc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7c5cb48d542f4cb3ab13c046c51b42dc 2023-04-03 02:55:20,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/07569886019c4db187084410ade7e28e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/07569886019c4db187084410ade7e28e 2023-04-03 02:55:20,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/61efeaaca5e249ac851c3c8a6f53ecd5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/61efeaaca5e249ac851c3c8a6f53ecd5 2023-04-03 02:55:20,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/ab07476b983c46eb8fe02a696068c5bd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/ab07476b983c46eb8fe02a696068c5bd 2023-04-03 02:55:20,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/bda2cd5ace9a4336a2b743214a5a9545 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/bda2cd5ace9a4336a2b743214a5a9545 2023-04-03 02:55:20,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/faee7a1237194568aa63bd94ea82b3bf to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/faee7a1237194568aa63bd94ea82b3bf 2023-04-03 02:55:20,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/3629a3bdf3bd4e8882b613c95b62d98e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/3629a3bdf3bd4e8882b613c95b62d98e 2023-04-03 02:55:20,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/5bf493a7c1a34d86a1722cc7d4c1209d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/5bf493a7c1a34d86a1722cc7d4c1209d 2023-04-03 02:55:20,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/142a7e82795244a7a6976ccd4482c505 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/142a7e82795244a7a6976ccd4482c505 2023-04-03 02:55:20,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/73ee2df950c64b9c910d99a5925cd877 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/73ee2df950c64b9c910d99a5925cd877 2023-04-03 02:55:20,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7e70cff08f754ac2a247a78d87d7d199 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7e70cff08f754ac2a247a78d87d7d199 2023-04-03 02:55:20,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/76436b28b8e54fff88f9ef510cb78a10 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/76436b28b8e54fff88f9ef510cb78a10 2023-04-03 02:55:20,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b3a42467107f47948d589ecea870418a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b3a42467107f47948d589ecea870418a 2023-04-03 02:55:20,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d5a384bed07f492abe1d99b1925aaca4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/d5a384bed07f492abe1d99b1925aaca4 2023-04-03 02:55:20,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0241ba6d0c6d4d31acce5550754cf4e8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0241ba6d0c6d4d31acce5550754cf4e8 2023-04-03 02:55:20,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4d4fbf8bb05e4c229708cbf0fe7b35a3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4d4fbf8bb05e4c229708cbf0fe7b35a3 2023-04-03 02:55:20,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6f841e787f95452dbfac980f1b5e4f4b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6f841e787f95452dbfac980f1b5e4f4b 2023-04-03 02:55:20,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/ac40fb1251e94acb8079874d09505110 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/ac40fb1251e94acb8079874d09505110 2023-04-03 02:55:20,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c61798973a7c4a70b3684e4f1fe38fdd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c61798973a7c4a70b3684e4f1fe38fdd 2023-04-03 02:55:20,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b174603456a34e36930b26da989ee291 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/b174603456a34e36930b26da989ee291 2023-04-03 02:55:20,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/a5b3b76c61b04bb696a372814202584a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/a5b3b76c61b04bb696a372814202584a 2023-04-03 02:55:20,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/70eefa62ea5d4d71a5808bc9bee77857 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/70eefa62ea5d4d71a5808bc9bee77857 2023-04-03 02:55:20,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/577b6b46ad4c4042b7466e3aba241654 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/577b6b46ad4c4042b7466e3aba241654 2023-04-03 02:55:20,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/52a188bb271842f2a3ab1d6b1d02d155 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/52a188bb271842f2a3ab1d6b1d02d155 2023-04-03 02:55:20,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/bb373193ab834c828792b0d9b1f02eec to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/bb373193ab834c828792b0d9b1f02eec 2023-04-03 02:55:20,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7d35d5adc73149f184bc53ba56bb488f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/7d35d5adc73149f184bc53ba56bb488f 2023-04-03 02:55:20,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cac489b1dd7d4eff96382dab41f31e24 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cac489b1dd7d4eff96382dab41f31e24 2023-04-03 02:55:20,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4b66b8305991458794251db6eefb1b8d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/4b66b8305991458794251db6eefb1b8d 2023-04-03 02:55:20,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/f80f5b86c6a04496b6635113af3c57d8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/f80f5b86c6a04496b6635113af3c57d8 2023-04-03 02:55:20,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/2f5616f3ae2343049f576b0fd8ec768c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/2f5616f3ae2343049f576b0fd8ec768c 2023-04-03 02:55:20,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cf4f213735744522b2cfb2b88d9e2b68 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cf4f213735744522b2cfb2b88d9e2b68 2023-04-03 02:55:20,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/99eea9ecebd04fe5b3b263345a352da6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/99eea9ecebd04fe5b3b263345a352da6 2023-04-03 02:55:20,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/e9a6e45dfec0453ba9561777f16fc1fa to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/e9a6e45dfec0453ba9561777f16fc1fa 2023-04-03 02:55:20,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cc16b7829ea347f4b2363fc9df3e8fb6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/cc16b7829ea347f4b2363fc9df3e8fb6 2023-04-03 02:55:20,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c40ec9439b6c4d50bc1633d9540c007b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/c40ec9439b6c4d50bc1633d9540c007b 2023-04-03 02:55:20,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1bb9a9b46dae4e578aa9a4c77fa73f30, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/fadc9e7128dd41ce97e571a469974cea, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/6030c50e962a4f96be0167dd704e9d9b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/2ac6fcd768e64fd6874d94fe1acf1a1a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e2e0f9750ccb435f9083f7e5805b12c9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/18842a11cbd6493fbd45879afcc889ba, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/41023184e1eb4148b71d01851c798aac, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/5a7fcbab4c434806990ce75ff7408ddd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e3e352849b774c23af3952af4d75c86e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8f90f1049d4c4d36a2404ea8e5bc9522, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c452c5be024148728de001c9c6be0f12, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/b29c3f5baa384856996d692edfd00e4a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f194788520544a24bd0af09f8965d0ae, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/a987b5bd18ff42448ab3297a58a77757, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/dec7ca8cf4dd4cae9c44f0077290f0b3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8fd622ecd11841a08758393b1a3144d7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/7c8012f1fd304bf999e840cfa781543d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ee2ac2d488f466689a6c9da31e91aa9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9a2f6c414f514606a0a7eaa217cca75e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/15e9a48e5f3a49639bc26be87772b564, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/4d5665842112474cb7f54d7579aaa408, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/360dc7073b7e4a5381f6124aab180189, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/ddbf48134e7e4cbfb1d99bd9629bae73, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8e39a052981348ac8e64965f8d080409, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f5a640372b174939a327a4bde6fe09ac, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f17266d17a454603bb818386e274810d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/da3d3951b4234af3b96ba92da09a154e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ef434d09075496eb7b2afdc69f62b93, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/242e30fadd724f489088286c01c7ef60, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f506142ad46549ef800af7ed37ea9738, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e403b02e4e3f425794b3f27a786f4af1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/d588af922f7f422883cc9a357688fb8b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/886ccf3bda82444c9c86c16328cb7858, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1e1ce09dc0954bf5bd30ccf6ad3f3e04, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f028ce727ff44ca592e1254492448634, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9c338ec7481745c5a3b89437753af0e7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c2e12daa0cdd4ce1895dd7e12ea7d2d0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/809786cd50bd40f79a0b3cf2735fc7fc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/601265f85f424658b8b66f1757c42875, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/05858b1ae8fd4249b750d0e6a26c2716, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/52abd4dfed1a4c36add13d700268a46d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/6dfd29ebd0ac4c31b820f281bbc35437, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/aa4300019a7a426c818742635b1b51d0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/72ca0a8b316349ca9c2bf4d01b7d1c87, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f8e058c1af63442d810748f04aef5938, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/311eb4b61d5b42f2bd38d4fc26754dbd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/559edb8a4b1b49ed9bc0e4c3da2152fc] to archive 2023-04-03 02:55:20,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:55:20,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1bb9a9b46dae4e578aa9a4c77fa73f30 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1bb9a9b46dae4e578aa9a4c77fa73f30 2023-04-03 02:55:20,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/fadc9e7128dd41ce97e571a469974cea to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/fadc9e7128dd41ce97e571a469974cea 2023-04-03 02:55:20,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/6030c50e962a4f96be0167dd704e9d9b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/6030c50e962a4f96be0167dd704e9d9b 2023-04-03 02:55:20,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/2ac6fcd768e64fd6874d94fe1acf1a1a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/2ac6fcd768e64fd6874d94fe1acf1a1a 2023-04-03 02:55:20,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e2e0f9750ccb435f9083f7e5805b12c9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e2e0f9750ccb435f9083f7e5805b12c9 2023-04-03 02:55:20,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/18842a11cbd6493fbd45879afcc889ba to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/18842a11cbd6493fbd45879afcc889ba 2023-04-03 02:55:20,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/41023184e1eb4148b71d01851c798aac to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/41023184e1eb4148b71d01851c798aac 2023-04-03 02:55:20,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/5a7fcbab4c434806990ce75ff7408ddd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/5a7fcbab4c434806990ce75ff7408ddd 2023-04-03 02:55:20,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e3e352849b774c23af3952af4d75c86e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e3e352849b774c23af3952af4d75c86e 2023-04-03 02:55:20,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8f90f1049d4c4d36a2404ea8e5bc9522 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8f90f1049d4c4d36a2404ea8e5bc9522 2023-04-03 02:55:20,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c452c5be024148728de001c9c6be0f12 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c452c5be024148728de001c9c6be0f12 2023-04-03 02:55:20,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/b29c3f5baa384856996d692edfd00e4a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/b29c3f5baa384856996d692edfd00e4a 2023-04-03 02:55:20,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f194788520544a24bd0af09f8965d0ae to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f194788520544a24bd0af09f8965d0ae 2023-04-03 02:55:20,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/a987b5bd18ff42448ab3297a58a77757 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/a987b5bd18ff42448ab3297a58a77757 2023-04-03 02:55:20,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/dec7ca8cf4dd4cae9c44f0077290f0b3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/dec7ca8cf4dd4cae9c44f0077290f0b3 2023-04-03 02:55:20,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8fd622ecd11841a08758393b1a3144d7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8fd622ecd11841a08758393b1a3144d7 2023-04-03 02:55:20,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/7c8012f1fd304bf999e840cfa781543d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/7c8012f1fd304bf999e840cfa781543d 2023-04-03 02:55:20,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ee2ac2d488f466689a6c9da31e91aa9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ee2ac2d488f466689a6c9da31e91aa9 2023-04-03 02:55:20,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9a2f6c414f514606a0a7eaa217cca75e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9a2f6c414f514606a0a7eaa217cca75e 2023-04-03 02:55:20,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/15e9a48e5f3a49639bc26be87772b564 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/15e9a48e5f3a49639bc26be87772b564 2023-04-03 02:55:20,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/4d5665842112474cb7f54d7579aaa408 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/4d5665842112474cb7f54d7579aaa408 2023-04-03 02:55:20,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/360dc7073b7e4a5381f6124aab180189 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/360dc7073b7e4a5381f6124aab180189 2023-04-03 02:55:20,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/ddbf48134e7e4cbfb1d99bd9629bae73 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/ddbf48134e7e4cbfb1d99bd9629bae73 2023-04-03 02:55:20,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8e39a052981348ac8e64965f8d080409 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8e39a052981348ac8e64965f8d080409 2023-04-03 02:55:20,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f5a640372b174939a327a4bde6fe09ac to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f5a640372b174939a327a4bde6fe09ac 2023-04-03 02:55:20,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f17266d17a454603bb818386e274810d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f17266d17a454603bb818386e274810d 2023-04-03 02:55:20,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/da3d3951b4234af3b96ba92da09a154e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/da3d3951b4234af3b96ba92da09a154e 2023-04-03 02:55:20,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ef434d09075496eb7b2afdc69f62b93 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9ef434d09075496eb7b2afdc69f62b93 2023-04-03 02:55:20,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/242e30fadd724f489088286c01c7ef60 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/242e30fadd724f489088286c01c7ef60 2023-04-03 02:55:20,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f506142ad46549ef800af7ed37ea9738 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f506142ad46549ef800af7ed37ea9738 2023-04-03 02:55:20,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e403b02e4e3f425794b3f27a786f4af1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/e403b02e4e3f425794b3f27a786f4af1 2023-04-03 02:55:20,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/d588af922f7f422883cc9a357688fb8b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/d588af922f7f422883cc9a357688fb8b 2023-04-03 02:55:20,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/886ccf3bda82444c9c86c16328cb7858 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/886ccf3bda82444c9c86c16328cb7858 2023-04-03 02:55:20,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1e1ce09dc0954bf5bd30ccf6ad3f3e04 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1e1ce09dc0954bf5bd30ccf6ad3f3e04 2023-04-03 02:55:20,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f028ce727ff44ca592e1254492448634 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f028ce727ff44ca592e1254492448634 2023-04-03 02:55:20,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9c338ec7481745c5a3b89437753af0e7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/9c338ec7481745c5a3b89437753af0e7 2023-04-03 02:55:20,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c2e12daa0cdd4ce1895dd7e12ea7d2d0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/c2e12daa0cdd4ce1895dd7e12ea7d2d0 2023-04-03 02:55:20,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/809786cd50bd40f79a0b3cf2735fc7fc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/809786cd50bd40f79a0b3cf2735fc7fc 2023-04-03 02:55:20,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/601265f85f424658b8b66f1757c42875 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/601265f85f424658b8b66f1757c42875 2023-04-03 02:55:20,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/05858b1ae8fd4249b750d0e6a26c2716 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/05858b1ae8fd4249b750d0e6a26c2716 2023-04-03 02:55:20,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/52abd4dfed1a4c36add13d700268a46d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/52abd4dfed1a4c36add13d700268a46d 2023-04-03 02:55:20,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/6dfd29ebd0ac4c31b820f281bbc35437 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/6dfd29ebd0ac4c31b820f281bbc35437 2023-04-03 02:55:20,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/aa4300019a7a426c818742635b1b51d0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/aa4300019a7a426c818742635b1b51d0 2023-04-03 02:55:20,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/72ca0a8b316349ca9c2bf4d01b7d1c87 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/72ca0a8b316349ca9c2bf4d01b7d1c87 2023-04-03 02:55:20,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f8e058c1af63442d810748f04aef5938 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f8e058c1af63442d810748f04aef5938 2023-04-03 02:55:20,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/311eb4b61d5b42f2bd38d4fc26754dbd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/311eb4b61d5b42f2bd38d4fc26754dbd 2023-04-03 02:55:20,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/559edb8a4b1b49ed9bc0e4c3da2152fc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/559edb8a4b1b49ed9bc0e4c3da2152fc 2023-04-03 02:55:20,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/recovered.edits/726.seqid, newMaxSeqId=726, maxSeqId=1 2023-04-03 02:55:20,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369. 2023-04-03 02:55:20,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ca59c59c9ebb324c8ea1a6deda6d6369: 2023-04-03 02:55:20,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:20,106 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=ca59c59c9ebb324c8ea1a6deda6d6369, regionState=CLOSED 2023-04-03 02:55:20,106 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490520106"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490520106"}]},"ts":"1680490520106"} 2023-04-03 02:55:20,108 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-03 02:55:20,108 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure ca59c59c9ebb324c8ea1a6deda6d6369, server=jenkins-hbase4.apache.org,43815,1680490421348 in 1.5680 sec 2023-04-03 02:55:20,109 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-03 02:55:20,109 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ca59c59c9ebb324c8ea1a6deda6d6369, UNASSIGN in 1.5710 sec 2023-04-03 02:55:20,110 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490520110"}]},"ts":"1680490520110"} 2023-04-03 02:55:20,111 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 02:55:20,112 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 02:55:20,113 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5810 sec 2023-04-03 02:55:20,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 02:55:20,638 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-03 02:55:20,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 02:55:20,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:20,641 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:20,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-03 02:55:20,642 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:20,647 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:20,649 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/recovered.edits] 2023-04-03 02:55:20,651 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/42392e45efa549c6bf9eff7267673ac2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/42392e45efa549c6bf9eff7267673ac2 2023-04-03 02:55:20,652 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b7fd6a886c544ce3b339ab394116126c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/b7fd6a886c544ce3b339ab394116126c 2023-04-03 02:55:20,653 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cde4628cf7564791b5d033ba01f9e144 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/A/cde4628cf7564791b5d033ba01f9e144 2023-04-03 02:55:20,655 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0e217be2ad274049846d594679776950 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/0e217be2ad274049846d594679776950 2023-04-03 02:55:20,657 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/3ab3a0e4603a46369866976d72aad31d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/3ab3a0e4603a46369866976d72aad31d 2023-04-03 02:55:20,658 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6f7d5fed1620412daac1f4df69465267 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/B/6f7d5fed1620412daac1f4df69465267 2023-04-03 02:55:20,661 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1ca3718667f045ceb170e54b52425fbf to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/1ca3718667f045ceb170e54b52425fbf 2023-04-03 02:55:20,662 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8515d105ddc644b8bb8fa4150d0a4a9e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/8515d105ddc644b8bb8fa4150d0a4a9e 2023-04-03 02:55:20,663 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f90dde3b400a45d4afd431e5cc1e59c2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/C/f90dde3b400a45d4afd431e5cc1e59c2 2023-04-03 02:55:20,666 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/recovered.edits/726.seqid to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369/recovered.edits/726.seqid 2023-04-03 02:55:20,667 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/ca59c59c9ebb324c8ea1a6deda6d6369 2023-04-03 02:55:20,667 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:55:20,669 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:20,670 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 02:55:20,672 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 02:55:20,672 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:20,672 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 02:55:20,673 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680490520672"}]},"ts":"9223372036854775807"} 2023-04-03 02:55:20,674 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 02:55:20,674 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ca59c59c9ebb324c8ea1a6deda6d6369, NAME => 'TestAcidGuarantees,,1680490487191.ca59c59c9ebb324c8ea1a6deda6d6369.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 02:55:20,674 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 02:55:20,674 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680490520674"}]},"ts":"9223372036854775807"} 2023-04-03 02:55:20,675 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 02:55:20,676 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:20,677 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 37 msec 2023-04-03 02:55:20,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-03 02:55:20,742 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-03 02:55:20,754 INFO [Listener at localhost/46853] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 224), OpenFileDescriptor=513 (was 507) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=398 (was 396) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=3395 (was 3589) 2023-04-03 02:55:20,764 INFO [Listener at localhost/46853] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=513, MaxFileDescriptor=60000, SystemLoadAverage=398, ProcessCount=171, AvailableMemoryMB=3395 2023-04-03 02:55:20,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 02:55:20,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 02:55:20,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:20,768 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 02:55:20,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-03 02:55:20,768 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 02:55:20,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 02:55:20,769 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:20,770 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c empty. 2023-04-03 02:55:20,770 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:20,770 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:55:20,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 02:55:21,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 02:55:21,177 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 02:55:21,179 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 21fe738407d44fd93b895283a93c3a0c, NAME => 'TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp 2023-04-03 02:55:21,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 02:55:21,586 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:55:21,587 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 21fe738407d44fd93b895283a93c3a0c, disabling compactions & flushes 2023-04-03 02:55:21,587 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:21,587 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:21,587 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. after waiting 0 ms 2023-04-03 02:55:21,587 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:21,587 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:21,587 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:21,589 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 02:55:21,589 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490521589"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490521589"}]},"ts":"1680490521589"} 2023-04-03 02:55:21,590 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 02:55:21,591 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 02:55:21,591 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490521591"}]},"ts":"1680490521591"} 2023-04-03 02:55:21,592 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 02:55:21,595 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=21fe738407d44fd93b895283a93c3a0c, ASSIGN}] 2023-04-03 02:55:21,597 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=21fe738407d44fd93b895283a93c3a0c, ASSIGN 2023-04-03 02:55:21,597 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=21fe738407d44fd93b895283a93c3a0c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43815,1680490421348; forceNewPlan=false, retain=false 2023-04-03 02:55:21,748 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=21fe738407d44fd93b895283a93c3a0c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:21,748 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490521748"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490521748"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490521748"}]},"ts":"1680490521748"} 2023-04-03 02:55:21,750 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:55:21,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 02:55:21,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:21,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 21fe738407d44fd93b895283a93c3a0c, NAME => 'TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:55:21,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:21,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:55:21,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:21,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:21,905 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:21,907 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A 2023-04-03 02:55:21,907 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A 2023-04-03 02:55:21,907 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:55:21,907 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 21fe738407d44fd93b895283a93c3a0c columnFamilyName A 2023-04-03 02:55:21,908 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(310): Store=21fe738407d44fd93b895283a93c3a0c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:55:21,908 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:21,909 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B 2023-04-03 02:55:21,909 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B 2023-04-03 02:55:21,909 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:55:21,909 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 21fe738407d44fd93b895283a93c3a0c columnFamilyName B 2023-04-03 02:55:21,910 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(310): Store=21fe738407d44fd93b895283a93c3a0c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:55:21,910 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:21,911 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C 2023-04-03 02:55:21,911 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C 2023-04-03 02:55:21,911 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:55:21,911 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 21fe738407d44fd93b895283a93c3a0c columnFamilyName C 2023-04-03 02:55:21,911 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(310): Store=21fe738407d44fd93b895283a93c3a0c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:55:21,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:21,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:21,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:21,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 02:55:21,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:21,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 02:55:21,916 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 21fe738407d44fd93b895283a93c3a0c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59235957, jitterRate=-0.11731545627117157}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 02:55:21,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:21,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., pid=37, masterSystemTime=1680490521901 2023-04-03 02:55:21,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:21,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:21,918 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=21fe738407d44fd93b895283a93c3a0c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:21,918 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490521918"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680490521918"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680490521918"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680490521918"}]},"ts":"1680490521918"} 2023-04-03 02:55:21,920 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-03 02:55:21,920 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 in 170 msec 2023-04-03 02:55:21,921 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-03 02:55:21,921 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=21fe738407d44fd93b895283a93c3a0c, ASSIGN in 325 msec 2023-04-03 02:55:21,922 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 02:55:21,922 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490521922"}]},"ts":"1680490521922"} 2023-04-03 02:55:21,923 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 02:55:21,924 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 02:55:21,925 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1580 sec 2023-04-03 02:55:22,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 02:55:22,873 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-03 02:55:22,874 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x6434ce29 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:22,878 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41507d16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:22,879 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:22,881 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45790, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:22,882 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 02:55:22,883 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59710, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 02:55:22,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 02:55:22,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 02:55:22,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:23,294 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 02:55:23,295 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 02:55:23,295 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 02:55:23,297 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-03 02:55:23,300 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=21fe738407d44fd93b895283a93c3a0c, REOPEN/MOVE}] 2023-04-03 02:55:23,300 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=21fe738407d44fd93b895283a93c3a0c, REOPEN/MOVE 2023-04-03 02:55:23,301 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=21fe738407d44fd93b895283a93c3a0c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:23,301 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490523301"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490523301"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490523301"}]},"ts":"1680490523301"} 2023-04-03 02:55:23,302 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:55:23,453 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 21fe738407d44fd93b895283a93c3a0c, disabling compactions & flushes 2023-04-03 02:55:23,454 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:23,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:23,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. after waiting 0 ms 2023-04-03 02:55:23,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:23,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-03 02:55:23,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:23,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:23,458 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 21fe738407d44fd93b895283a93c3a0c to self. 2023-04-03 02:55:23,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,459 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=21fe738407d44fd93b895283a93c3a0c, regionState=CLOSED 2023-04-03 02:55:23,460 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490523459"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490523459"}]},"ts":"1680490523459"} 2023-04-03 02:55:23,461 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-03 02:55:23,461 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 in 158 msec 2023-04-03 02:55:23,462 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=21fe738407d44fd93b895283a93c3a0c, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,43815,1680490421348; forceNewPlan=false, retain=true 2023-04-03 02:55:23,612 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=21fe738407d44fd93b895283a93c3a0c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:23,612 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490523612"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490523612"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490523612"}]},"ts":"1680490523612"} 2023-04-03 02:55:23,614 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:55:23,767 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:23,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 21fe738407d44fd93b895283a93c3a0c, NAME => 'TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:55:23,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:55:23,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,769 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,770 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A 2023-04-03 02:55:23,770 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A 2023-04-03 02:55:23,771 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:55:23,771 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 21fe738407d44fd93b895283a93c3a0c columnFamilyName A 2023-04-03 02:55:23,772 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(310): Store=21fe738407d44fd93b895283a93c3a0c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:55:23,772 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,773 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B 2023-04-03 02:55:23,773 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B 2023-04-03 02:55:23,773 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:55:23,773 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 21fe738407d44fd93b895283a93c3a0c columnFamilyName B 2023-04-03 02:55:23,774 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(310): Store=21fe738407d44fd93b895283a93c3a0c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:55:23,774 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,774 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C 2023-04-03 02:55:23,774 DEBUG [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C 2023-04-03 02:55:23,774 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:55:23,775 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 21fe738407d44fd93b895283a93c3a0c columnFamilyName C 2023-04-03 02:55:23,775 INFO [StoreOpener-21fe738407d44fd93b895283a93c3a0c-1] regionserver.HStore(310): Store=21fe738407d44fd93b895283a93c3a0c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:55:23,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:23,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 02:55:23,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 21fe738407d44fd93b895283a93c3a0c; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74364929, jitterRate=0.1081237941980362}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 02:55:23,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:23,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., pid=42, masterSystemTime=1680490523765 2023-04-03 02:55:23,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:23,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:23,782 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=21fe738407d44fd93b895283a93c3a0c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:23,782 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490523782"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680490523782"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680490523782"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680490523782"}]},"ts":"1680490523782"} 2023-04-03 02:55:23,783 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-03 02:55:23,784 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 in 169 msec 2023-04-03 02:55:23,784 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-03 02:55:23,785 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=21fe738407d44fd93b895283a93c3a0c, REOPEN/MOVE in 484 msec 2023-04-03 02:55:23,786 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-03 02:55:23,786 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-03 02:55:23,787 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 901 msec 2023-04-03 02:55:23,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-03 02:55:23,789 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x0bb27fd4 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:23,793 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b5bd0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:23,793 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x445919fd to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:23,798 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c8d927d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:23,798 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x4d010713 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:23,801 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@419b69bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:23,802 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x69100738 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:23,805 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c6cac96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:23,805 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x6f603c27 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:23,810 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a0df34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:23,810 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:23,815 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:23,815 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x21128196 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:23,819 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4366fcdd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:23,819 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e86d9 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:23,823 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a35a28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:23,824 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x711ebaf2 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:23,826 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1d4246, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:23,827 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x0a7503d4 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:23,831 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eb8f142, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:23,836 DEBUG [hconnection-0x7b7a6540-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:23,836 DEBUG [hconnection-0x1376e3fb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:23,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:23,837 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:23,844 DEBUG [hconnection-0x6af5646-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:23,845 DEBUG [hconnection-0x14e1631d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:23,845 DEBUG [hconnection-0x72f19221-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:23,845 DEBUG [hconnection-0x3f634443-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:23,845 DEBUG [hconnection-0x733838d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:23,845 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58172, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:23,847 DEBUG [hconnection-0x1adeda5c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:23,847 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58208, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:23,848 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58200, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:23,848 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58236, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:23,849 DEBUG [hconnection-0x3cb255ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:23,855 DEBUG [hconnection-0x60dfa95b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:23,856 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58246, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:23,857 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58262, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:23,858 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58188, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:23,858 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58220, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:23,859 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58252, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:23,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:23,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:23,864 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:23,864 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:23,864 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:23,865 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:23,865 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:23,865 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:23,869 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:23,869 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:23,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:23,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:23,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:23,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:23,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:23,871 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:23,871 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:23,871 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:23,871 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:23,871 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:23,871 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:23,871 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:23,871 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:23,871 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:23,873 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-03 02:55:23,875 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:23,875 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:23,875 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:23,875 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:23,875 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:23,875 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:23,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:23,877 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:23,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1680490523881, max timestamp=1680490523894Num uniques 859; 2023-04-03 02:55:23,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1680490523881, max timestamp=1680490523894Num uniques 859; 2023-04-03 02:55:23,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-03 02:55:23,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-03 02:55:23,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490523891, max timestamp=1680490523894Num uniques 859; 2023-04-03 02:55:23,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:23,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:23,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490583908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:23,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:23,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490583909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:23,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:23,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490583909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:23,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:23,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490583910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:23,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:23,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490583911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490584011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490584011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490584011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490584011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490584012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490584220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490584220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490584221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490584222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490584223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,329 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403cdef45d524b34a5baa7383199d11c592 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403cdef45d524b34a5baa7383199d11c592 2023-04-03 02:55:24,334 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/73f4204cb0164b748bcf0b0ae48822e9 2023-04-03 02:55:24,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490584524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490584526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490584526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490584529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:24,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490584532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,744 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/7711936759004045a44e3aebba1a18c5 2023-04-03 02:55:24,758 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/b940e36245ac4f518ccf3c261af0e0b9 2023-04-03 02:55:24,761 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/73f4204cb0164b748bcf0b0ae48822e9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/73f4204cb0164b748bcf0b0ae48822e9 2023-04-03 02:55:24,765 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/73f4204cb0164b748bcf0b0ae48822e9, entries=150, sequenceid=13, filesize=25.1 K 2023-04-03 02:55:24,766 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/7711936759004045a44e3aebba1a18c5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7711936759004045a44e3aebba1a18c5 2023-04-03 02:55:24,773 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7711936759004045a44e3aebba1a18c5, entries=150, sequenceid=13, filesize=11.6 K 2023-04-03 02:55:24,774 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/b940e36245ac4f518ccf3c261af0e0b9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/b940e36245ac4f518ccf3c261af0e0b9 2023-04-03 02:55:24,778 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/b940e36245ac4f518ccf3c261af0e0b9, entries=150, sequenceid=13, filesize=11.6 K 2023-04-03 02:55:24,778 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=154.31 KB/158010 for 21fe738407d44fd93b895283a93c3a0c in 905ms, sequenceid=13, compaction requested=false 2023-04-03 02:55:24,779 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:24,779 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:24,779 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:24,780 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:24,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:24,782 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:24,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da756be[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:24,782 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:24,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:24,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:24,783 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:24,783 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:24,783 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:24,783 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,783 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:24,786 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:24,789 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:24,789 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:24,789 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:24,789 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:24,789 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:24,790 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:24,790 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:24,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:24,790 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:24,791 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:24,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:24,792 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:24,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:24,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:24,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:24,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:24,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:24,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:24,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:24,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:24,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,795 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:24,795 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:24,795 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:24,794 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:24,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:24,795 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:24,895 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:24,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:24,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:24,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:24,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:24,898 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:24,898 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:24,898 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:24,898 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:24,899 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:24,899 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:24,900 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:24,900 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:24,900 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:24,900 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:24,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:24,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:24,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:24,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:24,901 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:24,901 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:24,901 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:24,901 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:24,901 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:24,901 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:24,901 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:24,901 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-04-03 02:55:24,902 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:24,902 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:24,902 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:24,902 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:24,902 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:24,902 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:25,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:25,032 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:25,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490585036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490585036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490585039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490585039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490585040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490585143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490585144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490585147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,318 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304031751471f021c4663bfd5284520cf2873 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031751471f021c4663bfd5284520cf2873 2023-04-03 02:55:25,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490585349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490585349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490585355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,648 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 02:55:25,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490585652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490585653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:25,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490585658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:25,723 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/7207918e06634680af1e9f22089cac70 2023-04-03 02:55:26,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490586042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490586051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,134 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/1542bdac279943a3b9e58ef10043fd04 2023-04-03 02:55:26,152 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/69b07afd445e4c019c65718ff0e88f02 2023-04-03 02:55:26,155 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/7207918e06634680af1e9f22089cac70 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/7207918e06634680af1e9f22089cac70 2023-04-03 02:55:26,159 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/7207918e06634680af1e9f22089cac70, entries=150, sequenceid=49, filesize=25.1 K 2023-04-03 02:55:26,159 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/1542bdac279943a3b9e58ef10043fd04 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1542bdac279943a3b9e58ef10043fd04 2023-04-03 02:55:26,163 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1542bdac279943a3b9e58ef10043fd04, entries=150, sequenceid=49, filesize=11.6 K 2023-04-03 02:55:26,163 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/69b07afd445e4c019c65718ff0e88f02 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/69b07afd445e4c019c65718ff0e88f02 2023-04-03 02:55:26,168 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/69b07afd445e4c019c65718ff0e88f02, entries=150, sequenceid=49, filesize=11.6 K 2023-04-03 02:55:26,168 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~425.95 KB/436176, currentSize=33.54 KB/34350 for 21fe738407d44fd93b895283a93c3a0c in 1267ms, sequenceid=49, compaction requested=false 2023-04-03 02:55:26,168 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:26,169 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:26,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:26,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:26,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:26,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,172 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:26,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74775aee[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:26,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:26,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,175 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:26,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:26,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:26,175 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,175 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:26,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:26,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:26,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:26,179 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:26,179 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:26,179 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:26,179 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:26,179 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:26,180 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,180 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,180 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,181 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:26,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:26,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,187 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:26,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,187 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:26,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:26,190 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:55:26,191 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:26,191 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:26,191 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:26,191 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:26,191 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:26,191 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:26,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490586264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490586265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680490526251, max timestamp=1680490526270Num uniques 859; 2023-04-03 02:55:26,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490586271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 02:55:26,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680490526251, max timestamp=1680490526270Num uniques 859; 2023-04-03 02:55:26,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680490526251, max timestamp=1680490526270Num uniques 859; 2023-04-03 02:55:26,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 02:55:26,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 02:55:26,287 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:26,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:26,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:26,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:26,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:26,290 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:26,290 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,290 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:26,290 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:26,291 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,291 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,295 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:26,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:26,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:26,296 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,296 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:26,296 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:26,297 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,297 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:26,297 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:26,297 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,297 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:26,297 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:26,298 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,298 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,298 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,298 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:26,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:26,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@695367fe[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:26,300 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:26,300 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,304 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:26,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:26,306 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,306 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:26,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:26,309 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:26,310 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:26,310 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:26,310 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:26,310 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:26,311 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,311 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,311 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,311 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,313 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,314 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,314 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,314 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,315 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,315 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,315 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,315 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,315 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,315 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:26,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:26,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:26,315 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,316 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:26,316 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:26,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490586395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490586396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490586401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,416 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:26,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:26,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:26,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:26,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:26,419 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:26,419 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,419 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:26,419 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:26,420 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,420 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,422 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,422 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,423 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,423 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:26,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:26,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:26,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:26,424 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,424 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:26,424 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:26,424 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:26,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,425 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,426 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,426 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:26,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2c0b2b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:26,427 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:26,427 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:26,428 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,428 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,429 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:26,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:26,432 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:26,432 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:26,432 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:26,432 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:26,432 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:26,434 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,434 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,434 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,439 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,439 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,439 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,439 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:26,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:26,439 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,439 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,440 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:26,440 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:26,439 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:26,439 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,440 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,440 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490586502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490586502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490586508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,540 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:26,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:26,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:26,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:26,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:26,543 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:26,543 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,543 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:26,543 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:26,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,545 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,545 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,545 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,545 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:26,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:26,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:26,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:26,546 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,547 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:26,547 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:26,547 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:26,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,548 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:26,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd9ae76[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:26,552 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:26,552 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:26,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:26,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:26,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:26,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,557 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:26,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:26,560 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:26,560 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:26,560 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:26,560 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:26,560 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:26,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,562 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,564 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,567 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:26,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:26,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,568 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,568 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,568 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,568 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:26,568 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:26,570 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,570 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,604 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403feb55fbacfa54e39a54c78d394bcdfd1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403feb55fbacfa54e39a54c78d394bcdfd1 2023-04-03 02:55:26,668 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:26,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:26,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:26,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:26,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:26,672 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:26,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:26,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:26,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,674 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,675 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,675 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:26,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:26,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:26,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:26,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:26,676 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,676 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:26,676 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:26,676 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:26,681 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,682 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,682 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:26,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b72909[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:26,683 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:26,683 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,684 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,684 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:26,684 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:26,684 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:26,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:26,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:26,687 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:26,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:26,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:26,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:26,687 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:26,689 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,689 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,689 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,689 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,690 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,693 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,693 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,693 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,693 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,693 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:26,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:26,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:26,693 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,693 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,693 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,693 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,694 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,694 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:26,694 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:26,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490586706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490586706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:26,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490586713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,794 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:26,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:26,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:26,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:26,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:26,797 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:26,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:26,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:26,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,798 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,799 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,799 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:26,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:26,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:26,800 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,800 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:26,800 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:26,800 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:26,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,803 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,803 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,803 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:26,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:26,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5129c616[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:26,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:26,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,805 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:26,805 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,805 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:26,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:26,806 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:26,807 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:26,807 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,807 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,807 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:26,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:26,809 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:26,809 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:26,809 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:26,809 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:26,809 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:26,810 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,810 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,810 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,811 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,812 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:26,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:26,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,815 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,816 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,816 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:26,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,916 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:26,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:26,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:26,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:26,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:26,919 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:26,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:26,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:26,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,920 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:26,922 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,922 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:26,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:26,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:26,922 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,922 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:26,922 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:26,922 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:26,924 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:26,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bb6ea4b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:26,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:26,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,926 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,926 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:26,926 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:26,926 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:26,929 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:26,929 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,929 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,929 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:26,930 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:26,930 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:26,930 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:26,930 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:26,930 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:26,930 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:26,930 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:26,933 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:26,933 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,933 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:26,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,935 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,936 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:26,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:26,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:26,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:26,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:26,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:26,938 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:26,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:26,938 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:26,938 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:26,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:26,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:26,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:26,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,939 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:26,939 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,009 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/31da6a04eb684c9eaba3d6404f910aca 2023-04-03 02:55:27,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490587012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490587012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,018 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/1bb50fbf45b34b728a1ce56d95931fac 2023-04-03 02:55:27,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490587019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,038 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:27,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:27,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:27,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:27,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:27,041 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:27,041 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,041 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:27,041 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:27,042 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,042 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,043 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:27,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:27,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:27,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:27,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:27,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:27,047 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bbc4955[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:27,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:27,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,049 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:27,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:27,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:27,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,054 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:27,057 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:27,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:27,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:27,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:27,057 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:27,059 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,059 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,059 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,060 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:27,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,063 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,063 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:27,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,164 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:27,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:27,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:27,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:27,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:27,167 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:27,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:27,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:27,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,168 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,168 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,169 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,169 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:27,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:27,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:27,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:27,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:27,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:27,170 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,170 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:27,170 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:27,170 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,171 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:27,171 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:27,171 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,171 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:27,171 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:27,173 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:27,175 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:27,175 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@159d44a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:27,175 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:27,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:27,177 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:27,180 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:27,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:27,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:27,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:27,180 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:27,181 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,181 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,182 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,182 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:27,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:27,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:27,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,187 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:27,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,187 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,287 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:27,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:27,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:27,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:27,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:27,290 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:27,290 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,290 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:27,290 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:27,290 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,290 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,291 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,292 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,293 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:27,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:27,294 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,294 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:27,294 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:27,294 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:27,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:27,296 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,296 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,296 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,296 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:27,297 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:27,297 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e5fbb01[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:27,298 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:27,298 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,299 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:27,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,301 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:27,301 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,301 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:27,303 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:27,303 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:27,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:27,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:27,304 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:27,305 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,305 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,305 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,307 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,311 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,311 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,311 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,311 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:27,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,311 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,311 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,311 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:27,311 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:27,311 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:27,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,412 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:27,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:27,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:27,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:27,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:27,415 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:27,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:27,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:27,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,416 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,417 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,417 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,417 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,417 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:27,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:27,418 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:27,418 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,418 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:27,419 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,419 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:27,419 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:27,419 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:27,421 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,422 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,422 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:27,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12b7ee33[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:27,423 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:27,423 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,424 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:27,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:27,424 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,424 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:27,424 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:27,425 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,425 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:27,426 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,426 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/98fefc99af9c4cf49e655efa34cc8ad3 2023-04-03 02:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:27,428 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:27,428 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:27,428 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:27,428 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:27,428 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:27,430 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,430 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,430 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,431 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,432 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/31da6a04eb684c9eaba3d6404f910aca as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/31da6a04eb684c9eaba3d6404f910aca 2023-04-03 02:55:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,432 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,436 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/31da6a04eb684c9eaba3d6404f910aca, entries=250, sequenceid=60, filesize=38.7 K 2023-04-03 02:55:27,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/1bb50fbf45b34b728a1ce56d95931fac as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1bb50fbf45b34b728a1ce56d95931fac 2023-04-03 02:55:27,443 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,443 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,443 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,443 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,443 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,443 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,443 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:27,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:27,443 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,444 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,444 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,444 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:27,444 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:27,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1bb50fbf45b34b728a1ce56d95931fac, entries=150, sequenceid=60, filesize=11.6 K 2023-04-03 02:55:27,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/98fefc99af9c4cf49e655efa34cc8ad3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/98fefc99af9c4cf49e655efa34cc8ad3 2023-04-03 02:55:27,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/98fefc99af9c4cf49e655efa34cc8ad3, entries=150, sequenceid=60, filesize=11.6 K 2023-04-03 02:55:27,450 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-03 02:55:27,450 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 21fe738407d44fd93b895283a93c3a0c in 1260ms, sequenceid=60, compaction requested=true 2023-04-03 02:55:27,450 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:27,451 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:27,451 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:27,451 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:27,451 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:27,451 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:27,452 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:27,452 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:27,452 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/A is initiating minor compaction (all files) 2023-04-03 02:55:27,452 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/B is initiating minor compaction (all files) 2023-04-03 02:55:27,452 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/A in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,452 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/B in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,452 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/73f4204cb0164b748bcf0b0ae48822e9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/7207918e06634680af1e9f22089cac70, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/31da6a04eb684c9eaba3d6404f910aca] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=89.0 K 2023-04-03 02:55:27,452 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7711936759004045a44e3aebba1a18c5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1542bdac279943a3b9e58ef10043fd04, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1bb50fbf45b34b728a1ce56d95931fac] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=34.8 K 2023-04-03 02:55:27,452 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 73f4204cb0164b748bcf0b0ae48822e9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680490523868 2023-04-03 02:55:27,452 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7711936759004045a44e3aebba1a18c5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680490523868 2023-04-03 02:55:27,453 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 7207918e06634680af1e9f22089cac70, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680490523905 2023-04-03 02:55:27,453 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 1542bdac279943a3b9e58ef10043fd04, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680490523905 2023-04-03 02:55:27,453 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 31da6a04eb684c9eaba3d6404f910aca, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490525038 2023-04-03 02:55:27,453 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 1bb50fbf45b34b728a1ce56d95931fac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490525039 2023-04-03 02:55:27,459 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#B#compaction#348 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:27,462 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#A#compaction#347 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:27,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:27,525 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-03 02:55:27,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:27,526 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:27,526 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:27,526 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:27,526 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:27,526 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:27,544 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:27,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:27,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490587541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:27,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:27,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:27,547 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:27,547 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,547 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:27,547 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:27,548 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,548 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,549 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,549 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,549 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:27,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:27,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:27,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:27,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:27,551 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:27,551 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,551 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:27,551 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:27,551 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,551 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:27,551 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:27,551 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,551 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:27,551 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:27,552 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,552 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,552 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490587546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bb8fbd1[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:27,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:27,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,557 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,557 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:27,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490587546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,559 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,559 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:27,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:27,561 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:27,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:27,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:27,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:27,562 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:27,563 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,563 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,563 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,563 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,564 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,566 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,566 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,566 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:27,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:27,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:27,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,567 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:27,567 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:27,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490587647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490587654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490587658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,667 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:27,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:27,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:27,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:27,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:27,670 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:27,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:27,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:27,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,672 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,673 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,673 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:27,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:27,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:27,674 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,674 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:27,674 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:27,674 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:27,677 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,678 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:27,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@248c9968[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:27,679 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:27,679 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,680 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:27,680 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:27,680 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:27,681 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,681 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:27,682 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,682 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:27,682 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,682 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:27,685 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:27,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:27,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:27,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:27,685 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:27,687 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,687 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,687 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,689 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,690 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,693 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,693 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,694 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,694 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,695 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,695 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:27,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:27,695 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,695 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,695 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,695 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:27,695 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:27,795 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:27,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:27,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:27,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:27,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:27,798 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:27,798 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,798 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:27,798 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:27,798 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,799 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,800 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,800 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,800 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:27,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:27,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:27,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:27,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:27,801 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,801 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:27,801 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:27,801 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:27,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:27,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e8a94b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:27,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:27,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,805 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:27,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:27,805 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:27,805 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,806 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:27,807 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:27,807 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,807 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,807 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:27,809 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:27,809 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:27,809 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:27,809 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:27,810 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:27,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,811 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,813 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,819 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:27,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:27,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:27,819 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:27,819 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:27,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490587849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490587859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:27,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490587865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,874 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/9156bcc1ad6a4f7ebf28a43c75b5a697 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/9156bcc1ad6a4f7ebf28a43c75b5a697 2023-04-03 02:55:27,879 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/B of 21fe738407d44fd93b895283a93c3a0c into 9156bcc1ad6a4f7ebf28a43c75b5a697(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:27,879 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:27,879 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/B, priority=13, startTime=1680490527451; duration=0sec 2023-04-03 02:55:27,879 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:27,879 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:27,880 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:27,880 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/C is initiating minor compaction (all files) 2023-04-03 02:55:27,880 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/C in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,880 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/b940e36245ac4f518ccf3c261af0e0b9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/69b07afd445e4c019c65718ff0e88f02, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/98fefc99af9c4cf49e655efa34cc8ad3] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=34.8 K 2023-04-03 02:55:27,881 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b940e36245ac4f518ccf3c261af0e0b9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680490523868 2023-04-03 02:55:27,881 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 69b07afd445e4c019c65718ff0e88f02, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680490523905 2023-04-03 02:55:27,881 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 98fefc99af9c4cf49e655efa34cc8ad3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490525039 2023-04-03 02:55:27,896 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#C#compaction#350 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:27,919 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:27,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:27,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:27,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:27,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:27,922 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:27,922 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,922 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:27,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:27,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,925 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,926 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,926 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:27,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:27,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:27,928 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,928 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:27,928 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:27,928 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:27,929 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:27,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8301f3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:27,931 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:27,931 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,932 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,932 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:27,932 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,932 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:27,932 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:27,932 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:27,933 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,933 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:27,933 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,933 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:27,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:27,936 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:27,936 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:27,936 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:27,936 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:27,936 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:27,938 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403f619d7618e774e5ea35759568c6d9c63 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f619d7618e774e5ea35759568c6d9c63 2023-04-03 02:55:27,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,938 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,939 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:27,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:27,939 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,941 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:27,943 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:27,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:27,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:27,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:27,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:27,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:27,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:27,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:27,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:27,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:27,943 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:27,943 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:27,943 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:28,043 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:28,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:28,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:28,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:28,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:28,047 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:28,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:28,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:28,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,047 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,048 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,049 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,049 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:28,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:28,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:28,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:28,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:28,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:28,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:28,051 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,051 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:28,051 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:28,052 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:28,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:28,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:28,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:28,053 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,053 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:28,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46c91901[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:28,055 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:28,055 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,057 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:28,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:28,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:28,058 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,058 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:28,059 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:28,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:28,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490588055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,062 DEBUG [Thread-2470] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:28,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:28,063 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:28,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:28,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:28,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:28,063 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:28,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:28,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490588058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,065 DEBUG [Thread-2472] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:28,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,066 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,067 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,069 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,069 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,069 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,072 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:28,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:28,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,072 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:28,073 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:28,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:28,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:28,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490588162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,173 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:28,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:28,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:28,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:28,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:28,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:28,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490588171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,176 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:28,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:28,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:28,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:28,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490588175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,177 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,177 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,180 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,181 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,181 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:28,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:28,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:28,182 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,182 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:28,182 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:28,182 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:28,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:28,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,184 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,184 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,184 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:28,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce8f69[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:28,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:28,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:28,187 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,187 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:28,188 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:28,190 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:28,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:28,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:28,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:28,190 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:28,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,194 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,195 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,197 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,197 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,197 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,197 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:28,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:28,197 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,197 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,198 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:28,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,198 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,197 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,198 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:28,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:28,198 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,198 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,304 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:28,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:28,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:28,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:28,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:28,308 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:28,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:28,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:28,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,308 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,312 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,313 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,313 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:28,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:28,314 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,314 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:28,314 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:28,314 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:28,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,317 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,318 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/0b50c2e643a54f6499efaf46d035ad3d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/0b50c2e643a54f6499efaf46d035ad3d 2023-04-03 02:55:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,318 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/2cb843ee49c74a6a9b9def7740085b0a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/2cb843ee49c74a6a9b9def7740085b0a 2023-04-03 02:55:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:28,319 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eedc0a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:28,319 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,320 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:28,320 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:28,320 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:28,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:28,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:28,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,322 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/A of 21fe738407d44fd93b895283a93c3a0c into 2cb843ee49c74a6a9b9def7740085b0a(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:28,322 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:28,322 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/A, priority=13, startTime=1680490527451; duration=0sec 2023-04-03 02:55:28,322 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/C of 21fe738407d44fd93b895283a93c3a0c into 0b50c2e643a54f6499efaf46d035ad3d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:28,322 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,322 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:28,323 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/C, priority=13, startTime=1680490527451; duration=0sec 2023-04-03 02:55:28,323 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:28,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:28,325 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:28,325 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:28,325 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:28,325 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:28,325 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:28,327 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,327 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,327 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,327 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,328 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:28,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:28,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,331 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,331 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:28,331 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:28,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,344 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=95, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/f20a45480b3e4ccab3371704f1bc3993 2023-04-03 02:55:28,431 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:28,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:28,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:28,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:28,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:28,434 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:28,434 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,434 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:28,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:28,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,436 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:28,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:28,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:28,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:28,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:28,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:28,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:28,437 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,438 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:28,438 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:28,438 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:28,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,439 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,439 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:28,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7610d9fd[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:28,441 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:28,441 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,442 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:28,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:28,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:28,442 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,442 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:28,445 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:28,447 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:28,447 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:28,447 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:28,447 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:28,447 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:28,448 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,448 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,448 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,449 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,450 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:28,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:28,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,452 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,452 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:28,452 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:28,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:28,553 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:28,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:28,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:28,556 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:28,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:28,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:28,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:28,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:28,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:28,559 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:28,559 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,560 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:28,560 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:28,560 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:28,565 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:28,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@781fd72f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:28,565 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,566 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,567 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:28,567 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,567 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:28,567 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:28,567 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:28,568 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,568 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:28,568 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,568 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:28,570 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:28,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:28,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:28,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:28,570 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:28,572 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,572 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,572 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,572 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,573 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:28,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:28,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,580 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,580 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:28,580 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:28,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,581 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:28,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,581 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,581 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:28,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490588669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,680 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:28,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:28,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:28,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:28,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:28,684 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:28,684 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:28,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:28,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:28,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490588680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:28,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490588680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,689 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,689 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,689 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,689 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:28,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:28,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:28,690 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:28,690 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,690 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:28,691 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,691 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:28,691 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:28,691 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:28,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,695 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:28,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ad8ba82[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:28,697 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:28,697 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,698 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:28,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:28,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:28,698 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,699 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:28,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:28,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,701 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:28,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:28,704 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:28,704 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:28,704 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:28,704 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:28,704 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:28,709 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,709 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,709 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,712 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,715 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:28,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:28,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:28,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,715 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:28,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,715 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:28,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/a8261411851743ad97e85444682d7654 2023-04-03 02:55:28,815 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:28,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:28,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:28,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:28,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:28,818 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:28,818 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,818 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:28,818 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:28,819 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,819 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,820 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,820 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,820 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,820 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:28,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:28,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:28,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:28,821 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:28,821 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,821 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:28,821 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:28,821 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:28,821 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,821 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:28,821 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:28,822 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:28,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:28,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:28,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:28,823 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,823 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,823 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,823 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:28,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7824295f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:28,825 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:28,825 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,826 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:28,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:28,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:28,828 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:28,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:28,830 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:28,830 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:28,830 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:28,830 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:28,830 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:28,832 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,832 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,832 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,835 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,835 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,835 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,836 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,835 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:28,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,836 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,836 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,836 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:28,836 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:28,836 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,836 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:28,836 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,936 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:28,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:28,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:28,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:28,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:28,939 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:28,939 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,939 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:28,939 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:28,940 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,940 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,942 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,942 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,943 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:28,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:28,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:28,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:28,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:28,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:28,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,946 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,946 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@88bc897[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:28,947 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:28,947 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,948 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:28,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:28,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:28,948 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,948 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:28,949 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,949 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:28,949 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,949 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:28,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:28,951 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:28,951 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:28,951 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:28,951 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:28,951 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:28,954 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,954 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:28,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,954 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,954 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,955 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:28,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:28,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:28,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:28,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:28,958 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:28,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:28,958 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:28,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:28,958 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:28,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:28,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:28,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,058 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:29,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:29,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:29,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:29,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:29,061 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:29,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:29,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:29,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:29,062 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,062 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:29,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:29,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:29,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:29,065 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:29,065 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:29,065 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:29,065 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:29,066 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,066 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,067 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:29,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e2bc006[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:29,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,070 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:29,070 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,070 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:29,074 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:29,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,074 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,075 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:29,077 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:29,077 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:29,077 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:29,077 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:29,077 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:29,078 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,078 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:29,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,079 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:29,079 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,080 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,081 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,082 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:29,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:29,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:29,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:29,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:29,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:29,082 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:29,082 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:29,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:29,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:29,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/92276c25af4d47a79c4c17cdf8f02f91 2023-04-03 02:55:29,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/f20a45480b3e4ccab3371704f1bc3993 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/f20a45480b3e4ccab3371704f1bc3993 2023-04-03 02:55:29,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/f20a45480b3e4ccab3371704f1bc3993, entries=200, sequenceid=95, filesize=31.9 K 2023-04-03 02:55:29,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/a8261411851743ad97e85444682d7654 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/a8261411851743ad97e85444682d7654 2023-04-03 02:55:29,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/a8261411851743ad97e85444682d7654, entries=150, sequenceid=95, filesize=11.6 K 2023-04-03 02:55:29,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/92276c25af4d47a79c4c17cdf8f02f91 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/92276c25af4d47a79c4c17cdf8f02f91 2023-04-03 02:55:29,182 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/92276c25af4d47a79c4c17cdf8f02f91, entries=150, sequenceid=95, filesize=11.6 K 2023-04-03 02:55:29,182 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:29,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 21fe738407d44fd93b895283a93c3a0c in 1657ms, sequenceid=95, compaction requested=false 2023-04-03 02:55:29,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:29,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:29,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:29,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:29,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:29,185 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:29,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:29,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:29,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:29,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,188 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:29,188 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:29,188 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,188 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:29,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:29,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:29,190 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:29,190 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:29,190 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:55:29,190 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:29,190 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:29,190 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:29,190 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:29,190 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:29,190 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:29,212 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403636e583335f54a4db07b04419495d682 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403636e583335f54a4db07b04419495d682 2023-04-03 02:55:29,227 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=108, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/77ef0e9238a14dc6bb0734fb0e8d9d1d 2023-04-03 02:55:29,264 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/fa206e5ac2d545b394e0d62e672d4c3e 2023-04-03 02:55:29,279 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/ca4549fc71d2440aa5ad81368da69e61 2023-04-03 02:55:29,284 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/77ef0e9238a14dc6bb0734fb0e8d9d1d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/77ef0e9238a14dc6bb0734fb0e8d9d1d 2023-04-03 02:55:29,290 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/77ef0e9238a14dc6bb0734fb0e8d9d1d, entries=150, sequenceid=108, filesize=25.1 K 2023-04-03 02:55:29,290 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/fa206e5ac2d545b394e0d62e672d4c3e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/fa206e5ac2d545b394e0d62e672d4c3e 2023-04-03 02:55:29,294 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/fa206e5ac2d545b394e0d62e672d4c3e, entries=150, sequenceid=108, filesize=11.6 K 2023-04-03 02:55:29,302 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/ca4549fc71d2440aa5ad81368da69e61 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ca4549fc71d2440aa5ad81368da69e61 2023-04-03 02:55:29,309 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ca4549fc71d2440aa5ad81368da69e61, entries=150, sequenceid=108, filesize=11.6 K 2023-04-03 02:55:29,310 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 21fe738407d44fd93b895283a93c3a0c in 120ms, sequenceid=108, compaction requested=true 2023-04-03 02:55:29,310 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:29,310 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:29,310 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:29,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:29,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:29,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:29,312 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,312 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:29,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@416867eb[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:29,315 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:29,315 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,316 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:29,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:29,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,317 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:29,318 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:29,321 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:29,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:29,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:29,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:29,321 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:29,323 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,323 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,323 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:29,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,324 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,325 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:29,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,328 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:29,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:29,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:29,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,328 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:29,329 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:29,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:29,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:29,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:29,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:29,429 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:29,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:29,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:29,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:29,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:29,433 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:29,433 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:29,433 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:29,433 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:29,433 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,433 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,435 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:29,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:29,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:29,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:29,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:29,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:29,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:29,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:29,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:29,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:29,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:29,440 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:29,442 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@185fd048[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:29,444 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:29,444 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,446 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,446 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:29,446 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:29,446 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:29,446 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,446 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:29,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,447 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:29,450 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:29,450 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:29,450 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:29,450 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:29,450 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:29,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,452 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,454 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,455 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,457 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:29,457 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:29,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:29,457 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:29,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:29,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:29,457 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:29,457 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:29,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:29,465 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,465 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,465 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,465 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,465 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,465 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,465 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,557 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:29,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:29,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:29,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:29,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:29,561 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:29,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:29,562 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:29,562 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:29,562 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,562 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:29,564 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:29,565 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,565 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:29,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:29,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:29,566 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:29,566 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:29,566 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:29,566 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:29,566 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:29,566 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:29,567 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:29,567 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:29,567 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:29,569 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:29,572 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec66ca4[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:29,572 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:29,574 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,574 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:29,574 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:29,575 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:29,575 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:29,575 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:29,575 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:29,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:29,578 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:29,578 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:29,578 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:29,578 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:29,578 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:29,584 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,584 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,584 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:29,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,584 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:29,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,586 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:29,591 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:29,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:29,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:29,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:29,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:29,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:29,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:29,591 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:29,591 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:29,692 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:29,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:29,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:29,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:29,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:29,696 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:29,696 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:29,696 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:29,696 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:29,696 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,696 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,698 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:29,698 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:29,698 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:29,698 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:29,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:29,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:29,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:29,699 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:29,699 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:29,700 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:29,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:29,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:29,700 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-03 02:55:29,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:29,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:29,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:29,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:29,701 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:29,701 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:29,719 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304031fb66df2fd5440bcbd940d3af13eaed2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031fb66df2fd5440bcbd940d3af13eaed2 2023-04-03 02:55:29,724 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=114, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/1952b6ecd7a0499eb29cbf3116fbddaf 2023-04-03 02:55:29,733 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/68f438d4a39b44f48cbdacdce4024f75 2023-04-03 02:55:29,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:29,786 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:29,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490529813, max timestamp=1680490529825Num uniques 859; 2023-04-03 02:55:29,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:29,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490529813, max timestamp=1680490529825Num uniques 859; 2023-04-03 02:55:29,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490529813, max timestamp=1680490529825Num uniques 859; 2023-04-03 02:55:29,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:29,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:29,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490529856, max timestamp=1680490529869Num uniques 1018; 2023-04-03 02:55:29,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-03 02:55:29,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490529856, max timestamp=1680490529869Num uniques 1018; 2023-04-03 02:55:29,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-03 02:55:29,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490529859, max timestamp=1680490529869Num uniques 1018; 2023-04-03 02:55:29,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 02:55:29,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680490529889, max timestamp=1680490529908Num uniques 1127; 2023-04-03 02:55:29,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-03 02:55:29,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680490529889, max timestamp=1680490529908Num uniques 1027; 2023-04-03 02:55:29,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680490529889, max timestamp=1680490529908Num uniques 1127; 2023-04-03 02:55:29,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-03 02:55:29,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-03 02:55:29,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680490529937, max timestamp=1680490529955Num uniques 1136; 2023-04-03 02:55:29,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680490529937, max timestamp=1680490529955Num uniques 1136; 2023-04-03 02:55:29,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680490529937, max timestamp=1680490529955Num uniques 1136; 2023-04-03 02:55:29,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-04-03 02:55:29,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-04-03 02:55:29,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-04-03 02:55:30,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680490529989, max timestamp=1680490529997Num uniques 1195; 2023-04-03 02:55:30,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680490529989, max timestamp=1680490529997Num uniques 1195; 2023-04-03 02:55:30,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54720, new segment data size=8920, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-03 02:55:30,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680490529989, max timestamp=1680490529997Num uniques 1195; 2023-04-03 02:55:30,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54720, new segment data size=8920, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-03 02:55:30,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54720, new segment data size=8920, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-03 02:55:30,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680490530042, max timestamp=1680490530050Num uniques 1054; 2023-04-03 02:55:30,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680490530042, max timestamp=1680490530057Num uniques 1054; 2023-04-03 02:55:30,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-03 02:55:30,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-03 02:55:30,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680490530042, max timestamp=1680490530050Num uniques 1054; 2023-04-03 02:55:30,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-03 02:55:30,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680490530108, max timestamp=1680490530123Num uniques 1013; 2023-04-03 02:55:30,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-03 02:55:30,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680490530108, max timestamp=1680490530123Num uniques 1013; 2023-04-03 02:55:30,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680490530108, max timestamp=1680490530123Num uniques 1013; 2023-04-03 02:55:30,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-03 02:55:30,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-03 02:55:30,143 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/3415dc82f1694aafa564a41301dfe159 2023-04-03 02:55:30,148 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/1952b6ecd7a0499eb29cbf3116fbddaf as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/1952b6ecd7a0499eb29cbf3116fbddaf 2023-04-03 02:55:30,151 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/1952b6ecd7a0499eb29cbf3116fbddaf, entries=50, sequenceid=114, filesize=11.6 K 2023-04-03 02:55:30,152 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/68f438d4a39b44f48cbdacdce4024f75 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/68f438d4a39b44f48cbdacdce4024f75 2023-04-03 02:55:30,156 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/68f438d4a39b44f48cbdacdce4024f75, entries=50, sequenceid=114, filesize=7.0 K 2023-04-03 02:55:30,156 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/3415dc82f1694aafa564a41301dfe159 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/3415dc82f1694aafa564a41301dfe159 2023-04-03 02:55:30,161 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/3415dc82f1694aafa564a41301dfe159, entries=50, sequenceid=114, filesize=7.0 K 2023-04-03 02:55:30,162 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=120.76 KB/123660 for 21fe738407d44fd93b895283a93c3a0c in 462ms, sequenceid=114, compaction requested=true 2023-04-03 02:55:30,162 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:30,162 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:30,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:30,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:30,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:30,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:30,164 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,164 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ebdbe8a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:30,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:30,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:30,166 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=127.47 KB heapSize=355.31 KB 2023-04-03 02:55:30,167 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:30,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:30,167 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:30,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:30,167 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:30,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:30,169 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:30,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:30,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:30,169 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,169 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:30,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:30,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:30,170 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:30,170 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:30,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:30,173 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:30,173 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:30,173 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:30,173 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:30,173 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:30,174 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,174 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,174 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,175 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:30,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:30,178 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:30,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:30,179 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:30,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:30,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:30,179 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:30,179 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,179 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:30,179 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,179 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,179 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:30,179 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,184 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040387f4e56678b5406fb980f93954fea929 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040387f4e56678b5406fb980f93954fea929 2023-04-03 02:55:30,191 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/69dd57eee0f04bcbac753df67bdf6ed4 2023-04-03 02:55:30,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490590194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490590194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490590195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,279 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:30,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:30,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:30,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:30,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:30,284 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:30,284 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:30,284 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:30,284 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:30,284 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,284 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:30,285 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:30,286 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,286 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:30,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:30,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:30,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:30,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:30,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:30,287 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:30,292 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:30,292 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:30,292 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:30,292 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:30,293 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,293 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:30,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce8d589[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:30,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:30,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,298 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,299 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:30,299 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:30,299 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:30,299 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,299 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:30,300 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,300 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:30,300 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,300 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:30,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:30,302 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:30,302 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:30,302 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:30,302 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:30,302 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:30,303 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,303 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,303 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:30,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:30,309 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:30,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:30,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:30,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:30,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:30,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:30,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,309 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:30,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:30,309 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:30,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490590307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490590307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490590309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,410 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:30,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:30,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:30,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:30,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:30,412 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:30,412 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:30,412 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:30,412 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:30,413 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,413 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,414 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:30,414 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:30,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,415 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:30,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:30,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:30,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:30,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:30,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:30,417 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:30,417 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:30,417 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:30,417 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:30,418 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,418 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,418 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,419 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:30,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad39bcc[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:30,420 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:30,420 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,421 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:30,421 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,421 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:30,422 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:30,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:30,425 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:30,425 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:30,425 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:30,425 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:30,425 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:30,427 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,428 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,428 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,429 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,432 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,432 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:30,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:30,432 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,432 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:30,432 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,432 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,432 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:30,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:30,432 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:30,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:30,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:30,432 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,432 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:30,433 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:30,433 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:30,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490590527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490590527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490590527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,533 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:30,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:30,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:30,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:30,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:30,536 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:30,536 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:30,536 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:30,536 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:30,536 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,536 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,537 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:30,537 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:30,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,538 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:30,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:30,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:30,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:30,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:30,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:30,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:30,539 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:30,539 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:30,539 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:30,539 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:30,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:30,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:30,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:30,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:30,542 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,542 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,542 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,542 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:30,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:30,543 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:30,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d6b7e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:30,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,544 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:30,545 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,545 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:30,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:30,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:30,546 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:30,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:30,548 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:30,548 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:30,548 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:30,548 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:30,548 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:30,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,549 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:30,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:30,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,551 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,552 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:30,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:30,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,555 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,555 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,555 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,555 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,555 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,555 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:30,555 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:30,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:30,555 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:30,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:30,555 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:30,556 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:30,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:30,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:30,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:30,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:30,600 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/7fc94e9c7a054334b83280f85f378f9e 2023-04-03 02:55:30,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/fc00d851d4764f91813769a71d8710d7 2023-04-03 02:55:30,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/69dd57eee0f04bcbac753df67bdf6ed4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/69dd57eee0f04bcbac753df67bdf6ed4 2023-04-03 02:55:30,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/69dd57eee0f04bcbac753df67bdf6ed4, entries=200, sequenceid=251, filesize=32.1 K 2023-04-03 02:55:30,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/7fc94e9c7a054334b83280f85f378f9e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7fc94e9c7a054334b83280f85f378f9e 2023-04-03 02:55:30,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7fc94e9c7a054334b83280f85f378f9e, entries=150, sequenceid=251, filesize=11.8 K 2023-04-03 02:55:30,628 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/fc00d851d4764f91813769a71d8710d7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/fc00d851d4764f91813769a71d8710d7 2023-04-03 02:55:30,631 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/fc00d851d4764f91813769a71d8710d7, entries=150, sequenceid=251, filesize=11.8 K 2023-04-03 02:55:30,632 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.27 KB/363792, currentSize=60.38 KB/61830 for 21fe738407d44fd93b895283a93c3a0c in 466ms, sequenceid=251, compaction requested=true 2023-04-03 02:55:30,632 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:30,632 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:30,632 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:55:30,632 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:55:30,632 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:30,632 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:30,634 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 129023 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:55:30,634 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 54997 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:55:30,634 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/A is initiating minor compaction (all files) 2023-04-03 02:55:30,634 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/B is initiating minor compaction (all files) 2023-04-03 02:55:30,634 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/A in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:30,634 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/B in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:30,634 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/2cb843ee49c74a6a9b9def7740085b0a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/f20a45480b3e4ccab3371704f1bc3993, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/77ef0e9238a14dc6bb0734fb0e8d9d1d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/1952b6ecd7a0499eb29cbf3116fbddaf, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/69dd57eee0f04bcbac753df67bdf6ed4] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=126.0 K 2023-04-03 02:55:30,634 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/9156bcc1ad6a4f7ebf28a43c75b5a697, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/a8261411851743ad97e85444682d7654, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/fa206e5ac2d545b394e0d62e672d4c3e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/68f438d4a39b44f48cbdacdce4024f75, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7fc94e9c7a054334b83280f85f378f9e] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=53.7 K 2023-04-03 02:55:30,634 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 2cb843ee49c74a6a9b9def7740085b0a, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490525039 2023-04-03 02:55:30,635 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 9156bcc1ad6a4f7ebf28a43c75b5a697, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490525039 2023-04-03 02:55:30,635 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting f20a45480b3e4ccab3371704f1bc3993, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680490526394 2023-04-03 02:55:30,635 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a8261411851743ad97e85444682d7654, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680490526394 2023-04-03 02:55:30,635 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 77ef0e9238a14dc6bb0734fb0e8d9d1d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680490527526 2023-04-03 02:55:30,635 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting fa206e5ac2d545b394e0d62e672d4c3e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680490527526 2023-04-03 02:55:30,635 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 1952b6ecd7a0499eb29cbf3116fbddaf, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680490529692 2023-04-03 02:55:30,636 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 68f438d4a39b44f48cbdacdce4024f75, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680490529692 2023-04-03 02:55:30,636 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 69dd57eee0f04bcbac753df67bdf6ed4, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680490530151 2023-04-03 02:55:30,636 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7fc94e9c7a054334b83280f85f378f9e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680490530151 2023-04-03 02:55:30,645 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#B#compaction#363 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:30,648 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#A#compaction#362 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:30,656 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:30,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:30,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:30,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:30,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:30,664 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:30,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:30,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:30,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:30,665 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,665 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,666 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:30,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:30,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:30,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:30,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:30,668 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/fe613b54304f4532ab69aa070ff0e30d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/fe613b54304f4532ab69aa070ff0e30d 2023-04-03 02:55:30,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:30,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:30,668 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:30,668 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:30,668 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:30,672 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/B of 21fe738407d44fd93b895283a93c3a0c into fe613b54304f4532ab69aa070ff0e30d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:30,672 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:30,672 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:30,672 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:30,672 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:30,672 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:30,672 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/B, priority=11, startTime=1680490530632; duration=0sec 2023-04-03 02:55:30,672 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:30,672 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:55:30,672 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:55:30,672 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:30,673 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:30,673 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:30,673 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:30,673 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:30,673 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:30,674 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 54997 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:55:30,674 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/C is initiating minor compaction (all files) 2023-04-03 02:55:30,675 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/C in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:30,675 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/0b50c2e643a54f6499efaf46d035ad3d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/92276c25af4d47a79c4c17cdf8f02f91, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ca4549fc71d2440aa5ad81368da69e61, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/3415dc82f1694aafa564a41301dfe159, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/fc00d851d4764f91813769a71d8710d7] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=53.7 K 2023-04-03 02:55:30,676 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 0b50c2e643a54f6499efaf46d035ad3d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490525039 2023-04-03 02:55:30,676 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 92276c25af4d47a79c4c17cdf8f02f91, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680490526394 2023-04-03 02:55:30,677 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ca4549fc71d2440aa5ad81368da69e61, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680490527526 2023-04-03 02:55:30,677 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 3415dc82f1694aafa564a41301dfe159, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680490529692 2023-04-03 02:55:30,678 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting fc00d851d4764f91813769a71d8710d7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680490530151 2023-04-03 02:55:30,689 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#C#compaction#365 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:30,700 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/e7461cf2af6d4b61b999d92b274da97b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/e7461cf2af6d4b61b999d92b274da97b 2023-04-03 02:55:30,704 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/C of 21fe738407d44fd93b895283a93c3a0c into e7461cf2af6d4b61b999d92b274da97b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:30,704 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:30,704 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/C, priority=11, startTime=1680490530632; duration=0sec 2023-04-03 02:55:30,704 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:30,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:30,835 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:30,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490530845, max timestamp=1680490530871Num uniques 859; 2023-04-03 02:55:30,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490530845, max timestamp=1680490530871Num uniques 859; 2023-04-03 02:55:30,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:55:30,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:55:30,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490530845, max timestamp=1680490530871Num uniques 859; 2023-04-03 02:55:30,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:55:30,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58236 deadline: 1680490590898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490590904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:30,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:30,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58246 deadline: 1680490590904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:31,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490591005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:31,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490591005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:31,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490591005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,087 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/bbe5e75205e9479ba8d20f6010d3a70b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/bbe5e75205e9479ba8d20f6010d3a70b 2023-04-03 02:55:31,090 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/A of 21fe738407d44fd93b895283a93c3a0c into bbe5e75205e9479ba8d20f6010d3a70b(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:31,091 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:31,091 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/A, priority=11, startTime=1680490530632; duration=0sec 2023-04-03 02:55:31,091 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:31,094 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a51bb73b894849cabf398f0aa9c702c6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a51bb73b894849cabf398f0aa9c702c6 2023-04-03 02:55:31,101 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=264, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/c84a820481f041229a5fd50937e47258 2023-04-03 02:55:31,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:31,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490591210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:31,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58252 deadline: 1680490591210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:31,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490591210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,510 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/bd6a2741c69b42a3af53298fa2fae227 2023-04-03 02:55:31,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:31,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490591517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:31,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490591517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:31,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490591517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,919 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/8d96f24771044706b3a550bbab4ce997 2023-04-03 02:55:31,923 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/c84a820481f041229a5fd50937e47258 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c84a820481f041229a5fd50937e47258 2023-04-03 02:55:31,926 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c84a820481f041229a5fd50937e47258, entries=150, sequenceid=264, filesize=25.4 K 2023-04-03 02:55:31,927 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/bd6a2741c69b42a3af53298fa2fae227 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/bd6a2741c69b42a3af53298fa2fae227 2023-04-03 02:55:31,930 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/bd6a2741c69b42a3af53298fa2fae227, entries=150, sequenceid=264, filesize=11.9 K 2023-04-03 02:55:31,930 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/8d96f24771044706b3a550bbab4ce997 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/8d96f24771044706b3a550bbab4ce997 2023-04-03 02:55:31,933 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/8d96f24771044706b3a550bbab4ce997, entries=150, sequenceid=264, filesize=11.9 K 2023-04-03 02:55:31,934 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 21fe738407d44fd93b895283a93c3a0c in 1262ms, sequenceid=264, compaction requested=false 2023-04-03 02:55:31,934 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:31,934 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:31,934 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:31,934 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:31,934 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,934 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:31,935 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:31,936 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:31,936 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:31,936 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:31,937 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:31,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:31,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74b7dcfb[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:31,938 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:31,938 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:31,939 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:31,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:31,939 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:31,939 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:31,939 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:31,939 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,939 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:31,941 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,941 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:31,942 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:31,942 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:31,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:31,944 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:31,944 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:31,944 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:31,944 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:31,944 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:31,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:31,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:31,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:31,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:31,946 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:31,947 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:31,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:31,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:31,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:31,953 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:31,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:31,953 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:31,953 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:31,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:31,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:31,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:31,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:31,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:31,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:31,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:32,025 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-03 02:55:32,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:32,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:32,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:32,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:32,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:32,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:32,041 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a23ab04530e04d489d5ba6afedce789c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a23ab04530e04d489d5ba6afedce789c 2023-04-03 02:55:32,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490592039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490592041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490592046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,053 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:32,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:32,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:32,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:32,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:32,056 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:32,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:32,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:32,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,057 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:32,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:32,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:32,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:32,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:32,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:32,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:32,059 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,059 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:32,059 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:32,059 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:32,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:32,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:32,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:32,062 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,062 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,062 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,062 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:32,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:32,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ea90b79[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:32,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,064 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,064 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,064 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:32,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:32,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:32,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:32,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:32,065 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,066 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:32,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:32,068 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:32,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:32,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:32,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:32,068 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:32,069 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,069 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,069 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,070 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,072 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:32,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:32,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,074 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,074 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:32,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,074 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:32,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,074 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490592081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,085 DEBUG [Thread-2470] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:32,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490592104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,109 DEBUG [Thread-2472] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8196 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:32,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490592147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490592148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490592154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,174 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:32,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:32,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:32,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:32,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:32,178 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:32,178 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,178 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:32,178 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:32,179 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,179 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,183 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:32,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:32,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:32,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:32,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:32,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:32,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:32,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:32,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:32,185 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,185 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:32,185 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:32,185 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:32,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:32,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:32,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:32,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48c2f89b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:32,188 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:32,188 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,189 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:32,189 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:32,189 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:32,190 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:32,191 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:32,191 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,191 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,191 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:32,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:32,193 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:32,193 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:32,193 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:32,193 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:32,193 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:32,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,196 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,197 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:32,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:32,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,201 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:32,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,202 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:32,202 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:32,302 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:32,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:32,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:32,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:32,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:32,305 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:32,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:32,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:32,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,307 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,307 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,307 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,307 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:32,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:32,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:32,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:32,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:32,309 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,309 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:32,309 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:32,309 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:32,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:32,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:32,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:32,310 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,310 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,311 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:32,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24783c66[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:32,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:32,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,314 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:32,314 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,314 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:32,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:32,317 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:32,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:32,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:32,319 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:32,319 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:32,320 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:32,320 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:32,320 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:32,321 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,321 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,321 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,328 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,330 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,330 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,331 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:32,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:32,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,331 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:32,331 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:32,331 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:32,332 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490592350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490592350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490592356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,432 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:32,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:32,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:32,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:32,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:32,435 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:32,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:32,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:32,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,438 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,438 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,438 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,439 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:32,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:32,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:32,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:32,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:32,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:32,441 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,441 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:32,441 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:32,441 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:32,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:32,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,442 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,442 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:32,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:32,444 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:32,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee27c16[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:32,444 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,445 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:32,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:32,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:32,446 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,446 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:32,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,447 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:32,447 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,447 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:32,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:32,449 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:32,449 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:32,449 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:32,449 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:32,449 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:32,452 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=300, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/fa629e320328428e831f8633e8404a34 2023-04-03 02:55:32,453 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,453 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,453 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,454 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,455 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,458 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:32,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:32,458 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,458 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,458 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,458 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,458 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,458 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,458 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,458 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,458 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:32,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,463 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:32,463 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:32,563 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:32,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:32,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:32,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:32,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:32,567 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:32,567 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,568 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:32,568 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:32,568 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,568 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,569 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,569 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:32,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:32,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:32,576 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:32,576 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,576 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:32,577 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:32,577 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:32,577 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,577 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:32,577 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:32,577 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,577 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:32,577 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:32,577 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,577 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:32,577 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:32,582 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,583 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,583 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:32,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8764070[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:32,584 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:32,584 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,585 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,585 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,585 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:32,585 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:32,585 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:32,585 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,587 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:32,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:32,590 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:32,590 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:32,590 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:32,590 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:32,590 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:32,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,592 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,592 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,593 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,601 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,601 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,601 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,601 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,601 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:32,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:32,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,601 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,601 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,602 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:32,602 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:32,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,602 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,602 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490592656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490592658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:32,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490592665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,702 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:32,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:32,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:32,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:32,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:32,705 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:32,705 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,705 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:32,705 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:32,706 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,706 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,707 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,707 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,707 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,707 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:32,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:32,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:32,708 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:32,709 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,709 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:32,709 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:32,709 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:32,710 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,710 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,711 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,711 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:32,712 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@572f67c5[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:32,712 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,713 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:32,714 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,714 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:32,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,716 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:32,716 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,716 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:32,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:32,718 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:32,718 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:32,718 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:32,718 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:32,718 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:32,720 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,720 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,720 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,720 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,722 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:32,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:32,724 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,724 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,724 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,724 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:32,724 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,724 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,724 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,724 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,824 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:32,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:32,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:32,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:32,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:32,827 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:32,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:32,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:32,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,828 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,829 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,829 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,829 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,829 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:32,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:32,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:32,830 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,830 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:32,831 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:32,831 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:32,833 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,833 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,833 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,833 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:32,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d1fa4a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:32,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:32,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,835 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:32,836 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,836 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:32,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:32,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:32,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:32,839 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:32,839 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:32,839 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:32,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:32,840 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:32,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:32,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:32,841 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:32,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:32,851 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,851 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:32,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:32,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:32,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:32,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:32,852 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:32,852 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:32,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/4f84eb429a20408fa330c99057d0b73f 2023-04-03 02:55:32,875 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/79a9746925a64e3a952a0f0bad5bb38d 2023-04-03 02:55:32,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/fa629e320328428e831f8633e8404a34 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/fa629e320328428e831f8633e8404a34 2023-04-03 02:55:32,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/fa629e320328428e831f8633e8404a34, entries=200, sequenceid=300, filesize=32.3 K 2023-04-03 02:55:32,882 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/4f84eb429a20408fa330c99057d0b73f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/4f84eb429a20408fa330c99057d0b73f 2023-04-03 02:55:32,885 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/4f84eb429a20408fa330c99057d0b73f, entries=150, sequenceid=300, filesize=11.9 K 2023-04-03 02:55:32,886 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/79a9746925a64e3a952a0f0bad5bb38d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/79a9746925a64e3a952a0f0bad5bb38d 2023-04-03 02:55:32,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/79a9746925a64e3a952a0f0bad5bb38d, entries=150, sequenceid=300, filesize=11.9 K 2023-04-03 02:55:32,890 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 21fe738407d44fd93b895283a93c3a0c in 865ms, sequenceid=300, compaction requested=true 2023-04-03 02:55:32,890 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:32,890 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:32,891 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:32,891 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:32,891 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:32,891 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:32,891 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85302 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:32,892 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:32,892 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/A is initiating minor compaction (all files) 2023-04-03 02:55:32,892 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/B is initiating minor compaction (all files) 2023-04-03 02:55:32,892 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/A in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,892 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/B in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,892 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/bbe5e75205e9479ba8d20f6010d3a70b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c84a820481f041229a5fd50937e47258, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/fa629e320328428e831f8633e8404a34] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=83.3 K 2023-04-03 02:55:32,892 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/fe613b54304f4532ab69aa070ff0e30d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/bd6a2741c69b42a3af53298fa2fae227, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/4f84eb429a20408fa330c99057d0b73f] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=35.8 K 2023-04-03 02:55:32,892 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting bbe5e75205e9479ba8d20f6010d3a70b, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680490530151 2023-04-03 02:55:32,892 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting fe613b54304f4532ab69aa070ff0e30d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680490530151 2023-04-03 02:55:32,893 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c84a820481f041229a5fd50937e47258, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680490530192 2023-04-03 02:55:32,893 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting bd6a2741c69b42a3af53298fa2fae227, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680490530192 2023-04-03 02:55:32,893 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting fa629e320328428e831f8633e8404a34, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680490530897 2023-04-03 02:55:32,893 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 4f84eb429a20408fa330c99057d0b73f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680490530897 2023-04-03 02:55:32,912 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#B#compaction#372 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:32,925 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#A#compaction#371 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:32,952 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:32,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:32,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:32,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:32,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:32,956 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:32,956 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,956 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:32,956 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:32,956 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,957 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:32,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:32,958 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,958 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:32,958 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:32,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:32,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:32,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:32,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:32,959 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:32,959 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:32,960 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 02:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:33,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:33,164 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:33,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490593197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:33,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490533189, max timestamp=1680490533199Num uniques 859; 2023-04-03 02:55:33,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:55:33,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490533189, max timestamp=1680490533199Num uniques 859; 2023-04-03 02:55:33,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680490533189, max timestamp=1680490533199Num uniques 859; 2023-04-03 02:55:33,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:55:33,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:55:33,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490593239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:33,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490593240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:33,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490593301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:33,329 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/e9748ea89e9a4f3aaeba07b388be36b9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/e9748ea89e9a4f3aaeba07b388be36b9 2023-04-03 02:55:33,333 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/B of 21fe738407d44fd93b895283a93c3a0c into e9748ea89e9a4f3aaeba07b388be36b9(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:33,333 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:33,333 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/B, priority=13, startTime=1680490532890; duration=0sec 2023-04-03 02:55:33,333 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:33,333 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:33,334 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:33,334 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/C is initiating minor compaction (all files) 2023-04-03 02:55:33,334 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/C in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:33,335 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/e7461cf2af6d4b61b999d92b274da97b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/8d96f24771044706b3a550bbab4ce997, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/79a9746925a64e3a952a0f0bad5bb38d] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=35.8 K 2023-04-03 02:55:33,335 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e7461cf2af6d4b61b999d92b274da97b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680490530151 2023-04-03 02:55:33,335 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 8d96f24771044706b3a550bbab4ce997, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680490530192 2023-04-03 02:55:33,336 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 79a9746925a64e3a952a0f0bad5bb38d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680490530897 2023-04-03 02:55:33,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490593347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:33,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490593347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:33,364 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#C#compaction#374 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:55:33,364 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/712346ea572041ae96dca11f97363467 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/712346ea572041ae96dca11f97363467 2023-04-03 02:55:33,371 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/A of 21fe738407d44fd93b895283a93c3a0c into 712346ea572041ae96dca11f97363467(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:33,371 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:33,371 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/A, priority=13, startTime=1680490532890; duration=0sec 2023-04-03 02:55:33,371 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:33,385 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304035fac05cdf52d48fb8f85c9aabeb9e814 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035fac05cdf52d48fb8f85c9aabeb9e814 2023-04-03 02:55:33,387 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/17f8c565c0b243fc8cac69aee531c684 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/17f8c565c0b243fc8cac69aee531c684 2023-04-03 02:55:33,391 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/C of 21fe738407d44fd93b895283a93c3a0c into 17f8c565c0b243fc8cac69aee531c684(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:33,391 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:33,391 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/C, priority=13, startTime=1680490532891; duration=0sec 2023-04-03 02:55:33,391 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:33,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490593506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:33,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490593551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:33,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490593551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:33,799 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=311, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/6ae410153c794f52afba27cac80cb265 2023-04-03 02:55:33,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490593811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:33,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490593855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:33,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:33,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490593855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,209 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/8103570fb9c744a6891a834f6f7b39cf 2023-04-03 02:55:34,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:34,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490594319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:34,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490594360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:34,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490594360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,618 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/77c883f457994842b6eaf784838e7b3f 2023-04-03 02:55:34,622 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/6ae410153c794f52afba27cac80cb265 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/6ae410153c794f52afba27cac80cb265 2023-04-03 02:55:34,625 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/6ae410153c794f52afba27cac80cb265, entries=150, sequenceid=311, filesize=25.4 K 2023-04-03 02:55:34,626 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/8103570fb9c744a6891a834f6f7b39cf as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/8103570fb9c744a6891a834f6f7b39cf 2023-04-03 02:55:34,629 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/8103570fb9c744a6891a834f6f7b39cf, entries=150, sequenceid=311, filesize=11.9 K 2023-04-03 02:55:34,629 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/77c883f457994842b6eaf784838e7b3f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/77c883f457994842b6eaf784838e7b3f 2023-04-03 02:55:34,633 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/77c883f457994842b6eaf784838e7b3f, entries=150, sequenceid=311, filesize=11.9 K 2023-04-03 02:55:34,634 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 21fe738407d44fd93b895283a93c3a0c in 1675ms, sequenceid=311, compaction requested=false 2023-04-03 02:55:34,634 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:34,634 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:34,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:34,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:34,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:34,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:34,636 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:34,636 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:34,636 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:34,636 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:34,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:34,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:34,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:34,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fae32ca[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:34,638 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:34,638 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:34,639 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:34,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:34,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:34,639 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,639 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:34,644 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,644 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:34,644 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:34,644 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:34,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:34,646 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:34,646 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:34,646 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:34,646 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:34,646 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:34,648 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:34,648 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:34,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:34,648 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:34,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:34,648 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:34,650 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,652 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,652 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:34,652 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:34,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:34,652 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:34,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:34,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:34,652 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:34,652 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:34,653 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,653 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:34,653 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:34,653 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:34,653 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:34,653 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:34,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:34,753 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:34,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:34,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:34,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:34,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:34,756 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:34,756 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:34,756 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:34,756 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:34,756 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:34,756 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:34,757 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,757 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:34,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:34,758 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:34,758 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:34,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:34,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:34,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:34,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:34,759 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:34,759 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:34,759 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:34,759 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:34,759 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:34,759 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:34,759 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:34,759 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-03 02:55:34,759 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:34,759 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:34,759 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:34,760 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:34,760 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:34,760 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:34,773 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403d22f813f0e0d4bdc98868461d1c222db to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d22f813f0e0d4bdc98868461d1c222db 2023-04-03 02:55:35,178 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/02e47e13c6d2483da4b2b3986988c299 2023-04-03 02:55:35,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:35,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:35,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:35,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490595375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:35,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:35,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490595386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:35,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:35,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490595386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:35,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:35,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:35,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490595494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:35,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490595494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:35,602 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/321a2ee7b9314f509ac35585ee3163fd 2023-04-03 02:55:35,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:35,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490595703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:35,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:35,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490595703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:36,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490596007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:36,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490596007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,014 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/46bcd1e136f34535b97caad0e1b04000 2023-04-03 02:55:36,018 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/02e47e13c6d2483da4b2b3986988c299 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/02e47e13c6d2483da4b2b3986988c299 2023-04-03 02:55:36,021 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/02e47e13c6d2483da4b2b3986988c299, entries=150, sequenceid=348, filesize=25.4 K 2023-04-03 02:55:36,022 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/321a2ee7b9314f509ac35585ee3163fd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/321a2ee7b9314f509ac35585ee3163fd 2023-04-03 02:55:36,027 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/321a2ee7b9314f509ac35585ee3163fd, entries=150, sequenceid=348, filesize=11.9 K 2023-04-03 02:55:36,028 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/46bcd1e136f34535b97caad0e1b04000 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/46bcd1e136f34535b97caad0e1b04000 2023-04-03 02:55:36,032 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/46bcd1e136f34535b97caad0e1b04000, entries=150, sequenceid=348, filesize=11.9 K 2023-04-03 02:55:36,033 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 21fe738407d44fd93b895283a93c3a0c in 1274ms, sequenceid=348, compaction requested=true 2023-04-03 02:55:36,033 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:36,033 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:36,036 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:36,036 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:36,038 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:36,038 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:36,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:36,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75618e8f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:36,039 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:36,039 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:36,040 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:36,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:36,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:36,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,041 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:36,043 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:36,043 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:36,043 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,043 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:36,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:36,048 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:36,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:36,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:36,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:36,049 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:36,050 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:36,050 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:36,050 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:36,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:36,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:36,052 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:36,054 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:36,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:36,060 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:36,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:36,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:36,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:36,060 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:36,061 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:36,061 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,061 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:36,061 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:36,061 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:36,061 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:36,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:36,061 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:36,161 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:36,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:36,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:36,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:36,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:36,165 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:36,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:36,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:36,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:36,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:36,165 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:36,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,166 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:36,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:36,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:36,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:36,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:36,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:36,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:36,168 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:36,168 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:36,169 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:55:36,169 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:36,169 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:36,169 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:36,169 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:36,169 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:36,169 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:36,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:36,513 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:36,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:36,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490596559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:36,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490596562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,589 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304033b5ec7f2bcf544188f832f65d46bf33e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033b5ec7f2bcf544188f832f65d46bf33e 2023-04-03 02:55:36,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:36,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490596666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:36,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490596669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:36,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490596870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:36,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490596875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:36,994 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=360, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/5c08514f7abe409fa6ed46d6943766f9 2023-04-03 02:55:37,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:37,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490597177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:37,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490597180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,403 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/983b90435fcf44ddacda198b23a7acc5 2023-04-03 02:55:37,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:37,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490597406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,412 DEBUG [Thread-2468] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:37,414 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/062d74701e25432fa37cc870d7f0b02d 2023-04-03 02:55:37,418 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/5c08514f7abe409fa6ed46d6943766f9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/5c08514f7abe409fa6ed46d6943766f9 2023-04-03 02:55:37,421 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/5c08514f7abe409fa6ed46d6943766f9, entries=150, sequenceid=360, filesize=25.4 K 2023-04-03 02:55:37,422 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/983b90435fcf44ddacda198b23a7acc5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/983b90435fcf44ddacda198b23a7acc5 2023-04-03 02:55:37,428 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/983b90435fcf44ddacda198b23a7acc5, entries=150, sequenceid=360, filesize=11.9 K 2023-04-03 02:55:37,429 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/062d74701e25432fa37cc870d7f0b02d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/062d74701e25432fa37cc870d7f0b02d 2023-04-03 02:55:37,433 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/062d74701e25432fa37cc870d7f0b02d, entries=150, sequenceid=360, filesize=11.9 K 2023-04-03 02:55:37,434 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 21fe738407d44fd93b895283a93c3a0c in 1265ms, sequenceid=360, compaction requested=true 2023-04-03 02:55:37,434 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:37,434 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:37,434 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:37,434 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:37,434 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:37,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:37,438 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:37,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:37,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@595ccea9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:37,440 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:37,440 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:37,441 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:37,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:37,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:37,441 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,441 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:37,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:37,441 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:37,443 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:37,443 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:37,444 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:37,444 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:37,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:37,448 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:37,448 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:37,448 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:37,448 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:37,448 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:37,449 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:37,449 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:37,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:37,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:37,450 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:37,450 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:37,451 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:37,454 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:37,454 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:37,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:37,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:37,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:37,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:37,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:37,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:37,455 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:37,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:37,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:37,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:37,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:37,455 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:37,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:37,455 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:37,455 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:37,556 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:37,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:37,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:37,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:37,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:37,559 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:37,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:37,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:37,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:37,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:37,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:37,560 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,560 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:37,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:37,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:37,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:37,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:37,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:37,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:37,562 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:37,562 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:37,563 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 02:55:37,563 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:37,563 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:37,563 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:37,563 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:37,563 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:37,563 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:37,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:37,689 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:37,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:37,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490597708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:37,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490597712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:37,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490597813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:37,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490597821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:37,985 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403ad212405e2494c2b9843fa9df7c37291 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ad212405e2494c2b9843fa9df7c37291 2023-04-03 02:55:38,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:38,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490598018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:38,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:38,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490598024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:38,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:38,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490598327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:38,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:38,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490598333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:38,390 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=380, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/d42c2e1bf80349b48fc36917a079f617 2023-04-03 02:55:38,800 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/d1fea2c012f24514944a43be373e94e0 2023-04-03 02:55:38,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:38,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490598834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:38,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:38,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490598842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,209 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/e4ac0fc6725e4f56904ff2a04dcd086a 2023-04-03 02:55:39,213 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/d42c2e1bf80349b48fc36917a079f617 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d42c2e1bf80349b48fc36917a079f617 2023-04-03 02:55:39,216 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d42c2e1bf80349b48fc36917a079f617, entries=150, sequenceid=380, filesize=25.4 K 2023-04-03 02:55:39,217 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/d1fea2c012f24514944a43be373e94e0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/d1fea2c012f24514944a43be373e94e0 2023-04-03 02:55:39,220 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/d1fea2c012f24514944a43be373e94e0, entries=150, sequenceid=380, filesize=11.9 K 2023-04-03 02:55:39,221 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/e4ac0fc6725e4f56904ff2a04dcd086a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/e4ac0fc6725e4f56904ff2a04dcd086a 2023-04-03 02:55:39,223 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/e4ac0fc6725e4f56904ff2a04dcd086a, entries=150, sequenceid=380, filesize=11.9 K 2023-04-03 02:55:39,224 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 21fe738407d44fd93b895283a93c3a0c in 1662ms, sequenceid=380, compaction requested=true 2023-04-03 02:55:39,224 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:39,224 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:39,227 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,227 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:39,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:39,227 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,227 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:39,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:39,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:39,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:39,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:39,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e27c62[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:39,228 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:39,229 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,229 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,230 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:39,230 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:39,230 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:39,230 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,230 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:39,231 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:39,233 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:39,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:39,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:39,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:39,233 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:39,237 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,239 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,240 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,242 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:39,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:39,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:39,242 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:39,242 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:39,242 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,242 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,242 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,242 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,242 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:39,242 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:39,243 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,243 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:39,244 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:39,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:39,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:39,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:39,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:39,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:39,342 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:39,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:39,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:39,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:39,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:39,345 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:39,345 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:39,345 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:39,345 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:39,346 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,346 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:39,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:39,347 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:39,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:39,347 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,347 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:39,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:39,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:39,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:39,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:39,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:39,348 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:39,348 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:39,348 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:39,348 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:39,348 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:39,348 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:39,348 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:39,348 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:55:39,349 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:39,349 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:39,349 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:39,349 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:39,349 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:39,349 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:39,363 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403522866c661e1409a9d0a859e6b31a022 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403522866c661e1409a9d0a859e6b31a022 2023-04-03 02:55:39,367 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=392, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/eb4acb8b570b454aa2f3707f770b4b2c 2023-04-03 02:55:39,377 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/16449f5c92ee424590e5c92de426db22 2023-04-03 02:55:39,785 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/ddb5c9cb08894194a5d84ed122dd399d 2023-04-03 02:55:39,789 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/eb4acb8b570b454aa2f3707f770b4b2c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/eb4acb8b570b454aa2f3707f770b4b2c 2023-04-03 02:55:39,792 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/eb4acb8b570b454aa2f3707f770b4b2c, entries=150, sequenceid=392, filesize=25.4 K 2023-04-03 02:55:39,793 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/16449f5c92ee424590e5c92de426db22 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16449f5c92ee424590e5c92de426db22 2023-04-03 02:55:39,796 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16449f5c92ee424590e5c92de426db22, entries=150, sequenceid=392, filesize=11.9 K 2023-04-03 02:55:39,796 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/ddb5c9cb08894194a5d84ed122dd399d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ddb5c9cb08894194a5d84ed122dd399d 2023-04-03 02:55:39,800 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ddb5c9cb08894194a5d84ed122dd399d, entries=150, sequenceid=392, filesize=11.9 K 2023-04-03 02:55:39,800 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 21fe738407d44fd93b895283a93c3a0c in 452ms, sequenceid=392, compaction requested=true 2023-04-03 02:55:39,800 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:39,800 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:39,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:39,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:39,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:39,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:39,802 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:39,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,803 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:39,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:39,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae42a56[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:39,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:39,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,806 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,806 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:39,806 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:39,806 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:39,807 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,807 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:39,808 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:39,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:39,811 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:39,811 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:39,811 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:39,811 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:39,811 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:39,813 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,813 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:39,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,813 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:39,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:39,814 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,815 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:39,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:39,818 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:39,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:39,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:39,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:39,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:39,819 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:39,819 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:39,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:39,818 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:39,819 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:39,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:39,919 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:39,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:39,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:39,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:39,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:39,922 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:39,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:39,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:39,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:39,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:39,925 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:39,925 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:39,926 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:39,926 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:39,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:39,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:39,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:39,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:39,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:39,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:39,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:39,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:39,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:39,927 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:39,927 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:39,927 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-03 02:55:39,927 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:39,928 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:39,928 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:39,928 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:39,928 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:39,928 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:39,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:39,932 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:40,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490539982, max timestamp=1680490540001Num uniques 859; 2023-04-03 02:55:40,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490539982, max timestamp=1680490540001Num uniques 859; 2023-04-03 02:55:40,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490539982, max timestamp=1680490540001Num uniques 859; 2023-04-03 02:55:40,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:40,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:40,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:40,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:40,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490600070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:40,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490540031, max timestamp=1680490540069Num uniques 1068; 2023-04-03 02:55:40,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-03 02:55:40,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490540031, max timestamp=1680490540069Num uniques 1068; 2023-04-03 02:55:40,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-03 02:55:40,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490540031, max timestamp=1680490540069Num uniques 1068; 2023-04-03 02:55:40,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-03 02:55:40,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:40,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490600189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:40,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:40,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490600189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:40,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:40,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490600300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:40,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:40,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490600300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:40,340 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a04c1dd39b4d4b829dbe77ab4adb2f80 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a04c1dd39b4d4b829dbe77ab4adb2f80 2023-04-03 02:55:40,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:40,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490600506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:40,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:40,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490600506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:40,747 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=400, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/c739137b4fba46a2ae780659643820ef 2023-04-03 02:55:40,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:40,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490600809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:40,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:40,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490600810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,156 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/e2a79342f8fd4c09ab4c0fff1be0a73a 2023-04-03 02:55:41,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:41,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490601322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:41,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490601322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:41,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58246 deadline: 1680490601424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,432 DEBUG [Thread-2468] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:41,567 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/f5f1c603f8984342a17618b858a1f19c 2023-04-03 02:55:41,571 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/c739137b4fba46a2ae780659643820ef as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c739137b4fba46a2ae780659643820ef 2023-04-03 02:55:41,574 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c739137b4fba46a2ae780659643820ef, entries=250, sequenceid=400, filesize=39.2 K 2023-04-03 02:55:41,575 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/e2a79342f8fd4c09ab4c0fff1be0a73a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/e2a79342f8fd4c09ab4c0fff1be0a73a 2023-04-03 02:55:41,578 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/e2a79342f8fd4c09ab4c0fff1be0a73a, entries=150, sequenceid=400, filesize=11.9 K 2023-04-03 02:55:41,578 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/f5f1c603f8984342a17618b858a1f19c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f5f1c603f8984342a17618b858a1f19c 2023-04-03 02:55:41,580 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f5f1c603f8984342a17618b858a1f19c, entries=150, sequenceid=400, filesize=11.9 K 2023-04-03 02:55:41,581 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 21fe738407d44fd93b895283a93c3a0c in 1654ms, sequenceid=400, compaction requested=true 2023-04-03 02:55:41,581 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:41,581 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:41,581 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:41,581 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:41,581 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,581 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:41,581 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:41,583 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,583 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:41,583 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:41,583 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:41,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:41,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2308c273[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:41,584 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:41,584 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:41,585 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:41,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:41,586 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:41,586 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:41,586 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:41,586 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,586 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:41,587 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,587 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:41,587 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:41,587 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:41,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:41,589 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:41,589 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:41,589 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:41,589 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:41,589 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:41,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:41,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:41,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:41,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:41,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:41,591 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:41,592 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,594 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,594 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:41,594 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:41,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:41,594 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:41,594 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:41,594 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:41,594 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:41,594 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:41,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:41,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:41,594 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:41,595 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:41,595 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:41,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:41,595 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:41,695 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:41,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:41,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:41,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:41,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:41,698 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:41,698 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:41,698 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:41,698 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:41,699 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:41,699 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:41,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,700 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:41,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:41,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:41,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:41,700 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:41,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:41,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:41,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:41,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:41,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:41,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:41,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:41,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:41,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:41,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:41,702 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:41,702 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:41,702 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:41,702 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:41,702 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-03 02:55:41,702 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:41,702 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:41,702 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:41,702 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:41,702 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:41,702 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:42,114 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304034d1531a8499e4d309014976c5ded5c99 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034d1531a8499e4d309014976c5ded5c99 2023-04-03 02:55:42,118 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:42,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:42,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:42,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490602151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:42,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490602157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:42,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490602259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:42,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490602263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:42,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490602341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:42,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490602342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:42,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490602467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:42,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490602474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,520 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=454, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/88fec2bc3e6b4e10bce6e700662432f7 2023-04-03 02:55:42,533 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/ca0a09848b8e47dca7256573d105d2ea 2023-04-03 02:55:42,542 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/184ab9f6464140b78805d767dc6abfe1 2023-04-03 02:55:42,545 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/88fec2bc3e6b4e10bce6e700662432f7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/88fec2bc3e6b4e10bce6e700662432f7 2023-04-03 02:55:42,548 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/88fec2bc3e6b4e10bce6e700662432f7, entries=150, sequenceid=454, filesize=25.4 K 2023-04-03 02:55:42,549 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/ca0a09848b8e47dca7256573d105d2ea as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/ca0a09848b8e47dca7256573d105d2ea 2023-04-03 02:55:42,552 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/ca0a09848b8e47dca7256573d105d2ea, entries=150, sequenceid=454, filesize=11.9 K 2023-04-03 02:55:42,553 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/184ab9f6464140b78805d767dc6abfe1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/184ab9f6464140b78805d767dc6abfe1 2023-04-03 02:55:42,556 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/184ab9f6464140b78805d767dc6abfe1, entries=150, sequenceid=454, filesize=11.9 K 2023-04-03 02:55:42,556 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=33.54 KB/34350 for 21fe738407d44fd93b895283a93c3a0c in 854ms, sequenceid=454, compaction requested=true 2023-04-03 02:55:42,556 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:42,556 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:42,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:42,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:42,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,557 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:42,557 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:42,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,558 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:42,558 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:42,559 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:42,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:42,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c1e793d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:42,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:42,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:42,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:42,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:42,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:42,563 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:42,563 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:42,563 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,563 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:42,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:42,565 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:42,565 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:42,565 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:42,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:42,567 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:42,567 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:42,567 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:42,567 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:42,567 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:42,568 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:42,568 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:42,568 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:42,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:42,568 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:42,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:42,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,575 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,575 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:42,575 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:42,575 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:42,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:42,576 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:42,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:42,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:42,575 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:42,576 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,576 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:42,576 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:42,576 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:42,576 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:42,576 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:42,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:42,627 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/73f4204cb0164b748bcf0b0ae48822e9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/7207918e06634680af1e9f22089cac70, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/31da6a04eb684c9eaba3d6404f910aca, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/2cb843ee49c74a6a9b9def7740085b0a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/f20a45480b3e4ccab3371704f1bc3993, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/77ef0e9238a14dc6bb0734fb0e8d9d1d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/1952b6ecd7a0499eb29cbf3116fbddaf, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/69dd57eee0f04bcbac753df67bdf6ed4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/bbe5e75205e9479ba8d20f6010d3a70b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c84a820481f041229a5fd50937e47258, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/fa629e320328428e831f8633e8404a34] to archive 2023-04-03 02:55:42,632 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:55:42,633 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/73f4204cb0164b748bcf0b0ae48822e9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/73f4204cb0164b748bcf0b0ae48822e9 2023-04-03 02:55:42,635 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/7207918e06634680af1e9f22089cac70 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/7207918e06634680af1e9f22089cac70 2023-04-03 02:55:42,636 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/31da6a04eb684c9eaba3d6404f910aca to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/31da6a04eb684c9eaba3d6404f910aca 2023-04-03 02:55:42,637 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/2cb843ee49c74a6a9b9def7740085b0a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/2cb843ee49c74a6a9b9def7740085b0a 2023-04-03 02:55:42,638 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/f20a45480b3e4ccab3371704f1bc3993 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/f20a45480b3e4ccab3371704f1bc3993 2023-04-03 02:55:42,639 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/77ef0e9238a14dc6bb0734fb0e8d9d1d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/77ef0e9238a14dc6bb0734fb0e8d9d1d 2023-04-03 02:55:42,640 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/1952b6ecd7a0499eb29cbf3116fbddaf to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/1952b6ecd7a0499eb29cbf3116fbddaf 2023-04-03 02:55:42,641 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/69dd57eee0f04bcbac753df67bdf6ed4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/69dd57eee0f04bcbac753df67bdf6ed4 2023-04-03 02:55:42,642 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/bbe5e75205e9479ba8d20f6010d3a70b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/bbe5e75205e9479ba8d20f6010d3a70b 2023-04-03 02:55:42,643 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c84a820481f041229a5fd50937e47258 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c84a820481f041229a5fd50937e47258 2023-04-03 02:55:42,644 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/fa629e320328428e831f8633e8404a34 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/fa629e320328428e831f8633e8404a34 2023-04-03 02:55:42,665 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7711936759004045a44e3aebba1a18c5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1542bdac279943a3b9e58ef10043fd04, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/9156bcc1ad6a4f7ebf28a43c75b5a697, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1bb50fbf45b34b728a1ce56d95931fac, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/a8261411851743ad97e85444682d7654, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/fa206e5ac2d545b394e0d62e672d4c3e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/68f438d4a39b44f48cbdacdce4024f75, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/fe613b54304f4532ab69aa070ff0e30d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7fc94e9c7a054334b83280f85f378f9e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/bd6a2741c69b42a3af53298fa2fae227, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/4f84eb429a20408fa330c99057d0b73f] to archive 2023-04-03 02:55:42,666 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:55:42,668 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7711936759004045a44e3aebba1a18c5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7711936759004045a44e3aebba1a18c5 2023-04-03 02:55:42,669 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1542bdac279943a3b9e58ef10043fd04 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1542bdac279943a3b9e58ef10043fd04 2023-04-03 02:55:42,670 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/9156bcc1ad6a4f7ebf28a43c75b5a697 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/9156bcc1ad6a4f7ebf28a43c75b5a697 2023-04-03 02:55:42,670 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1bb50fbf45b34b728a1ce56d95931fac to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/1bb50fbf45b34b728a1ce56d95931fac 2023-04-03 02:55:42,672 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/a8261411851743ad97e85444682d7654 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/a8261411851743ad97e85444682d7654 2023-04-03 02:55:42,673 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/fa206e5ac2d545b394e0d62e672d4c3e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/fa206e5ac2d545b394e0d62e672d4c3e 2023-04-03 02:55:42,674 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/68f438d4a39b44f48cbdacdce4024f75 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/68f438d4a39b44f48cbdacdce4024f75 2023-04-03 02:55:42,675 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/fe613b54304f4532ab69aa070ff0e30d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/fe613b54304f4532ab69aa070ff0e30d 2023-04-03 02:55:42,676 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:42,676 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7fc94e9c7a054334b83280f85f378f9e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/7fc94e9c7a054334b83280f85f378f9e 2023-04-03 02:55:42,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:42,678 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/bd6a2741c69b42a3af53298fa2fae227 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/bd6a2741c69b42a3af53298fa2fae227 2023-04-03 02:55:42,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:42,679 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/4f84eb429a20408fa330c99057d0b73f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/4f84eb429a20408fa330c99057d0b73f 2023-04-03 02:55:42,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:42,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:42,681 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:42,681 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:42,682 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:42,682 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:42,682 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/b940e36245ac4f518ccf3c261af0e0b9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/69b07afd445e4c019c65718ff0e88f02, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/0b50c2e643a54f6499efaf46d035ad3d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/98fefc99af9c4cf49e655efa34cc8ad3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/92276c25af4d47a79c4c17cdf8f02f91, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ca4549fc71d2440aa5ad81368da69e61, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/3415dc82f1694aafa564a41301dfe159, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/e7461cf2af6d4b61b999d92b274da97b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/fc00d851d4764f91813769a71d8710d7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/8d96f24771044706b3a550bbab4ce997, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/79a9746925a64e3a952a0f0bad5bb38d] to archive 2023-04-03 02:55:42,683 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:55:42,684 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:42,684 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:42,684 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/b940e36245ac4f518ccf3c261af0e0b9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/b940e36245ac4f518ccf3c261af0e0b9 2023-04-03 02:55:42,685 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/69b07afd445e4c019c65718ff0e88f02 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/69b07afd445e4c019c65718ff0e88f02 2023-04-03 02:55:42,686 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/0b50c2e643a54f6499efaf46d035ad3d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/0b50c2e643a54f6499efaf46d035ad3d 2023-04-03 02:55:42,687 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/98fefc99af9c4cf49e655efa34cc8ad3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/98fefc99af9c4cf49e655efa34cc8ad3 2023-04-03 02:55:42,688 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/92276c25af4d47a79c4c17cdf8f02f91 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/92276c25af4d47a79c4c17cdf8f02f91 2023-04-03 02:55:42,689 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ca4549fc71d2440aa5ad81368da69e61 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ca4549fc71d2440aa5ad81368da69e61 2023-04-03 02:55:42,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:42,690 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:42,690 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/3415dc82f1694aafa564a41301dfe159 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/3415dc82f1694aafa564a41301dfe159 2023-04-03 02:55:42,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:42,692 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:42,692 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:42,692 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/e7461cf2af6d4b61b999d92b274da97b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/e7461cf2af6d4b61b999d92b274da97b 2023-04-03 02:55:42,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:42,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:42,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:42,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:42,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:42,693 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/fc00d851d4764f91813769a71d8710d7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/fc00d851d4764f91813769a71d8710d7 2023-04-03 02:55:42,694 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/8d96f24771044706b3a550bbab4ce997 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/8d96f24771044706b3a550bbab4ce997 2023-04-03 02:55:42,695 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/79a9746925a64e3a952a0f0bad5bb38d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/79a9746925a64e3a952a0f0bad5bb38d 2023-04-03 02:55:42,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:42,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:42,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:42,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:42,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:42,700 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:42,700 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:42,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:42,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:42,700 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-03 02:55:42,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:42,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:42,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:42,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:42,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:42,701 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:42,720 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403773f6b83c62f40098cfc281d820fd65e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403773f6b83c62f40098cfc281d820fd65e 2023-04-03 02:55:42,725 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=462, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/d68791f129484e519d7c052d2f8cceb1 2023-04-03 02:55:42,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:42,800 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:42,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490542802, max timestamp=1680490542904Num uniques 859; 2023-04-03 02:55:42,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490542802, max timestamp=1680490542904Num uniques 859; 2023-04-03 02:55:42,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490542802, max timestamp=1680490542904Num uniques 859; 2023-04-03 02:55:42,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:42,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:42,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:43,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490543016, max timestamp=1680490543050Num uniques 1018; 2023-04-03 02:55:43,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 02:55:43,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490543016, max timestamp=1680490543050Num uniques 1018; 2023-04-03 02:55:43,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 02:55:43,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490543016, max timestamp=1680490543050Num uniques 1018; 2023-04-03 02:55:43,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 02:55:43,142 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/16fa9c0748594af495dcca8f41ff099b 2023-04-03 02:55:43,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680490543114, max timestamp=1680490543146Num uniques 1027; 2023-04-03 02:55:43,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-03 02:55:43,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680490543114, max timestamp=1680490543146Num uniques 1027; 2023-04-03 02:55:43,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-03 02:55:43,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680490543114, max timestamp=1680490543146Num uniques 1027; 2023-04-03 02:55:43,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-03 02:55:43,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680490543213, max timestamp=1680490543238Num uniques 1086; 2023-04-03 02:55:43,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-03 02:55:43,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680490543213, max timestamp=1680490543238Num uniques 1086; 2023-04-03 02:55:43,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680490543213, max timestamp=1680490543238Num uniques 1086; 2023-04-03 02:55:43,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-03 02:55:43,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-03 02:55:43,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:43,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490603311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680490543284, max timestamp=1680490543321Num uniques 1095; 2023-04-03 02:55:43,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680490543284, max timestamp=1680490543321Num uniques 1095; 2023-04-03 02:55:43,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-03 02:55:43,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-03 02:55:43,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680490543284, max timestamp=1680490543321Num uniques 1095; 2023-04-03 02:55:43,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-03 02:55:43,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680490543413, max timestamp=1680490543471Num uniques 1054; 2023-04-03 02:55:43,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-03 02:55:43,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680490543413, max timestamp=1680490543471Num uniques 1104; 2023-04-03 02:55:43,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680490543413, max timestamp=1680490543471Num uniques 1104; 2023-04-03 02:55:43,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-03 02:55:43,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-03 02:55:43,553 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/f6955a89445145ce96c90b73170b9ca2 2023-04-03 02:55:43,556 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/d68791f129484e519d7c052d2f8cceb1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d68791f129484e519d7c052d2f8cceb1 2023-04-03 02:55:43,560 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d68791f129484e519d7c052d2f8cceb1, entries=150, sequenceid=462, filesize=25.4 K 2023-04-03 02:55:43,561 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/16fa9c0748594af495dcca8f41ff099b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16fa9c0748594af495dcca8f41ff099b 2023-04-03 02:55:43,563 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16fa9c0748594af495dcca8f41ff099b, entries=150, sequenceid=462, filesize=11.9 K 2023-04-03 02:55:43,564 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/f6955a89445145ce96c90b73170b9ca2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f6955a89445145ce96c90b73170b9ca2 2023-04-03 02:55:43,567 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f6955a89445145ce96c90b73170b9ca2, entries=150, sequenceid=462, filesize=11.9 K 2023-04-03 02:55:43,575 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=114.05 KB/116790 for 21fe738407d44fd93b895283a93c3a0c in 875ms, sequenceid=462, compaction requested=true 2023-04-03 02:55:43,575 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:43,575 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:43,575 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:43,575 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:43,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:43,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=120.76 KB heapSize=340.92 KB 2023-04-03 02:55:43,575 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,575 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:43,575 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:43,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:43,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:43,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:43,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:43,576 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:43,576 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:43,577 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:43,577 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:43,577 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,577 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:43,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:43,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b6807b0[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:43,578 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:43,579 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,580 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:43,580 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:43,580 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:43,580 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,580 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:43,582 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,582 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:43,583 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:43,583 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:43,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:43,585 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:43,585 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:43,585 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:43,585 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:43,585 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:43,587 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,587 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:43,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,587 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:43,587 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,589 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:43,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:43,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:43,591 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:43,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:43,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:43,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:43,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:43,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,591 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:43,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,591 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:43,591 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:43,592 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:43,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490603614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:43,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490603625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,692 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:43,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:43,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:43,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:43,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:43,695 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:43,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:43,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:43,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:43,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,696 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,696 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:43,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:43,697 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,697 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:43,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:43,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:43,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:43,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:43,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:43,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:43,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:43,698 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:43,698 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:43,698 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:43,698 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:43,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:43,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:43,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:43,698 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:43,701 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:43,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:43,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:43,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a53611[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:43,702 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:43,702 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,703 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:43,703 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,703 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:43,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:43,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:43,704 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,704 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:43,705 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:43,705 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:43,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:43,707 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:43,707 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:43,707 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:43,707 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:43,707 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:43,708 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,708 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:43,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,709 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:43,711 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,712 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:43,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:43,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:43,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:43,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:43,715 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:43,715 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:43,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:43,715 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:43,715 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:43,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:43,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490603729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490603746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,816 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:43,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:43,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:43,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:43,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:43,819 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:43,819 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:43,819 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:43,819 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:43,819 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,819 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,820 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,821 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:43,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:43,821 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,821 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:43,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:43,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:43,822 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:43,822 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. started... 2023-04-03 02:55:43,822 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. as already flushing 2023-04-03 02:55:43,822 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,824 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:43,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:43,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b652c[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:43,826 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:43,826 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,827 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:43,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:43,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:43,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:43,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:43,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:43,829 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:43,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:43,832 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:43,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:43,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:43,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:43,832 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:43,833 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,833 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:43,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,833 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:43,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:43,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,835 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,835 DEBUG [Thread-2479] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e86d9 to 127.0.0.1:52506 2023-04-03 02:55:43,835 DEBUG [Thread-2479] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:43,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,842 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:43,842 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:43,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:43,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:43,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:43,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:43,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:43,842 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:43,842 DEBUG [Thread-2474] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:43,842 DEBUG [Thread-2474] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:43,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:43,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:43,843 DEBUG [Thread-2483] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a7503d4 to 127.0.0.1:52506 2023-04-03 02:55:43,843 DEBUG [Thread-2483] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:43,844 DEBUG [Thread-2481] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x711ebaf2 to 127.0.0.1:52506 2023-04-03 02:55:43,844 DEBUG [Thread-2481] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:43,848 DEBUG [Thread-2477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21128196 to 127.0.0.1:52506 2023-04-03 02:55:43,849 DEBUG [Thread-2477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:43,857 DEBUG [Thread-2475] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:52506 2023-04-03 02:55:43,857 DEBUG [Thread-2475] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:43,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:43,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490603935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,942 DEBUG [Thread-2474] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:43,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:43,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:43,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490603959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:43,988 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403f3f0d5a2619545b586febd168c4e9392 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f3f0d5a2619545b586febd168c4e9392 2023-04-03 02:55:44,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:44,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490604239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:44,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:44,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490604260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:44,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:44,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58252 deadline: 1680490604357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:44,358 DEBUG [Thread-2464] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:44,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:44,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58236 deadline: 1680490604366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:44,368 DEBUG [Thread-2466] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:44,393 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=582, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/9171ff41f1f6450fad05fd273be403ab 2023-04-03 02:55:44,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:44,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58208 deadline: 1680490604741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:44,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:44,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58188 deadline: 1680490604763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:44,801 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/c41484f5105a4ec38fe474ca56b044cc 2023-04-03 02:55:45,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/154f33b1ec3a464e83441e687d004cbc 2023-04-03 02:55:45,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/9171ff41f1f6450fad05fd273be403ab as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/9171ff41f1f6450fad05fd273be403ab 2023-04-03 02:55:45,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/9171ff41f1f6450fad05fd273be403ab, entries=250, sequenceid=582, filesize=39.2 K 2023-04-03 02:55:45,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/c41484f5105a4ec38fe474ca56b044cc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/c41484f5105a4ec38fe474ca56b044cc 2023-04-03 02:55:45,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/c41484f5105a4ec38fe474ca56b044cc, entries=150, sequenceid=582, filesize=11.9 K 2023-04-03 02:55:45,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/154f33b1ec3a464e83441e687d004cbc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/154f33b1ec3a464e83441e687d004cbc 2023-04-03 02:55:45,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/154f33b1ec3a464e83441e687d004cbc, entries=150, sequenceid=582, filesize=11.9 K 2023-04-03 02:55:45,222 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.88 KB/349056, currentSize=60.38 KB/61830 for 21fe738407d44fd93b895283a93c3a0c in 1646ms, sequenceid=582, compaction requested=true 2023-04-03 02:55:45,222 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:45,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:45,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:45,222 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-03 02:55:45,222 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-03 02:55:45,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:45,224 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 289004 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-03 02:55:45,224 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 122236 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-03 02:55:45,224 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/A is initiating minor compaction (all files) 2023-04-03 02:55:45,224 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/B is initiating minor compaction (all files) 2023-04-03 02:55:45,224 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/A in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:45,224 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/B in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:45,224 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/712346ea572041ae96dca11f97363467, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/6ae410153c794f52afba27cac80cb265, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/02e47e13c6d2483da4b2b3986988c299, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/5c08514f7abe409fa6ed46d6943766f9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d42c2e1bf80349b48fc36917a079f617, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/eb4acb8b570b454aa2f3707f770b4b2c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c739137b4fba46a2ae780659643820ef, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/88fec2bc3e6b4e10bce6e700662432f7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d68791f129484e519d7c052d2f8cceb1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/9171ff41f1f6450fad05fd273be403ab] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=282.2 K 2023-04-03 02:55:45,224 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/e9748ea89e9a4f3aaeba07b388be36b9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/8103570fb9c744a6891a834f6f7b39cf, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/321a2ee7b9314f509ac35585ee3163fd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/983b90435fcf44ddacda198b23a7acc5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/d1fea2c012f24514944a43be373e94e0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16449f5c92ee424590e5c92de426db22, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/e2a79342f8fd4c09ab4c0fff1be0a73a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/ca0a09848b8e47dca7256573d105d2ea, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16fa9c0748594af495dcca8f41ff099b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/c41484f5105a4ec38fe474ca56b044cc] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=119.4 K 2023-04-03 02:55:45,224 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 712346ea572041ae96dca11f97363467, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680490530897 2023-04-03 02:55:45,224 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e9748ea89e9a4f3aaeba07b388be36b9, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680490530897 2023-04-03 02:55:45,225 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6ae410153c794f52afba27cac80cb265, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680490532038 2023-04-03 02:55:45,225 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 8103570fb9c744a6891a834f6f7b39cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680490532038 2023-04-03 02:55:45,225 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 02e47e13c6d2483da4b2b3986988c299, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680490533212 2023-04-03 02:55:45,225 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 321a2ee7b9314f509ac35585ee3163fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680490533212 2023-04-03 02:55:45,225 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 5c08514f7abe409fa6ed46d6943766f9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680490535363 2023-04-03 02:55:45,225 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 983b90435fcf44ddacda198b23a7acc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680490535363 2023-04-03 02:55:45,225 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting d42c2e1bf80349b48fc36917a079f617, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680490536549 2023-04-03 02:55:45,225 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d1fea2c012f24514944a43be373e94e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680490536549 2023-04-03 02:55:45,225 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting eb4acb8b570b454aa2f3707f770b4b2c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680490537697 2023-04-03 02:55:45,225 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 16449f5c92ee424590e5c92de426db22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680490537697 2023-04-03 02:55:45,226 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c739137b4fba46a2ae780659643820ef, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680490539890 2023-04-03 02:55:45,226 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e2a79342f8fd4c09ab4c0fff1be0a73a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680490539890 2023-04-03 02:55:45,226 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 88fec2bc3e6b4e10bce6e700662432f7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680490540129 2023-04-03 02:55:45,226 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ca0a09848b8e47dca7256573d105d2ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680490540129 2023-04-03 02:55:45,226 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting d68791f129484e519d7c052d2f8cceb1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1680490542130 2023-04-03 02:55:45,226 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 16fa9c0748594af495dcca8f41ff099b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1680490542130 2023-04-03 02:55:45,226 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 9171ff41f1f6450fad05fd273be403ab, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680490543483 2023-04-03 02:55:45,226 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting c41484f5105a4ec38fe474ca56b044cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680490543546 2023-04-03 02:55:45,237 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#B#compaction#402 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:45,239 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#A#compaction#401 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:45,645 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/f545e3c7b9af4a5c97c7dfe6e22fb69a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/f545e3c7b9af4a5c97c7dfe6e22fb69a 2023-04-03 02:55:45,649 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/B of 21fe738407d44fd93b895283a93c3a0c into f545e3c7b9af4a5c97c7dfe6e22fb69a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:45,649 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:45,649 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/B, priority=6, startTime=1680490545222; duration=0sec 2023-04-03 02:55:45,649 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:45,649 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-03 02:55:45,651 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 122236 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-03 02:55:45,651 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 21fe738407d44fd93b895283a93c3a0c/C is initiating minor compaction (all files) 2023-04-03 02:55:45,651 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 21fe738407d44fd93b895283a93c3a0c/C in TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:45,651 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/17f8c565c0b243fc8cac69aee531c684, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/77c883f457994842b6eaf784838e7b3f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/46bcd1e136f34535b97caad0e1b04000, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/062d74701e25432fa37cc870d7f0b02d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/e4ac0fc6725e4f56904ff2a04dcd086a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ddb5c9cb08894194a5d84ed122dd399d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f5f1c603f8984342a17618b858a1f19c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/184ab9f6464140b78805d767dc6abfe1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f6955a89445145ce96c90b73170b9ca2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/154f33b1ec3a464e83441e687d004cbc] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp, totalSize=119.4 K 2023-04-03 02:55:45,651 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 17f8c565c0b243fc8cac69aee531c684, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680490530897 2023-04-03 02:55:45,651 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 77c883f457994842b6eaf784838e7b3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680490532038 2023-04-03 02:55:45,651 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 46bcd1e136f34535b97caad0e1b04000, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680490533212 2023-04-03 02:55:45,652 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 062d74701e25432fa37cc870d7f0b02d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680490535363 2023-04-03 02:55:45,652 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e4ac0fc6725e4f56904ff2a04dcd086a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680490536549 2023-04-03 02:55:45,652 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ddb5c9cb08894194a5d84ed122dd399d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680490537697 2023-04-03 02:55:45,652 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f5f1c603f8984342a17618b858a1f19c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680490539890 2023-04-03 02:55:45,652 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 184ab9f6464140b78805d767dc6abfe1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680490540129 2023-04-03 02:55:45,653 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f6955a89445145ce96c90b73170b9ca2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1680490542130 2023-04-03 02:55:45,653 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 154f33b1ec3a464e83441e687d004cbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680490543546 2023-04-03 02:55:45,663 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 21fe738407d44fd93b895283a93c3a0c#C#compaction#403 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:45,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:45,750 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 02:55:45,750 DEBUG [Thread-2470] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69100738 to 127.0.0.1:52506 2023-04-03 02:55:45,750 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:45,750 DEBUG [Thread-2470] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:45,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:45,750 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:45,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:45,750 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:45,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:45,772 DEBUG [Thread-2472] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f603c27 to 127.0.0.1:52506 2023-04-03 02:55:45,772 DEBUG [Thread-2472] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:46,070 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/cebefe00c6714caf84690bdae382ded0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/cebefe00c6714caf84690bdae382ded0 2023-04-03 02:55:46,073 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/C of 21fe738407d44fd93b895283a93c3a0c into cebefe00c6714caf84690bdae382ded0(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:46,073 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:46,073 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/C, priority=6, startTime=1680490545222; duration=0sec 2023-04-03 02:55:46,073 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:46,159 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403d1a5ccf1254c40e394bf3e8d0003fb26 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d1a5ccf1254c40e394bf3e8d0003fb26 2023-04-03 02:55:46,454 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/445d919ff0044c82b292cab9854f4033 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/445d919ff0044c82b292cab9854f4033 2023-04-03 02:55:46,457 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in 21fe738407d44fd93b895283a93c3a0c/A of 21fe738407d44fd93b895283a93c3a0c into 445d919ff0044c82b292cab9854f4033(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-03 02:55:46,457 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:46,458 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c., storeName=21fe738407d44fd93b895283a93c3a0c/A, priority=6, startTime=1680490545222; duration=1sec 2023-04-03 02:55:46,458 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:46,564 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=596, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/105b319be47c4b0697ff66f8bedc8af5 2023-04-03 02:55:46,972 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=596 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/17f21c2702a54d10808b6446b01c038a 2023-04-03 02:55:47,380 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=596 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/4fafa47b2083432a872091def5cc0c60 2023-04-03 02:55:47,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/105b319be47c4b0697ff66f8bedc8af5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/105b319be47c4b0697ff66f8bedc8af5 2023-04-03 02:55:47,386 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/105b319be47c4b0697ff66f8bedc8af5, entries=150, sequenceid=596, filesize=25.4 K 2023-04-03 02:55:47,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/17f21c2702a54d10808b6446b01c038a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/17f21c2702a54d10808b6446b01c038a 2023-04-03 02:55:47,390 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/17f21c2702a54d10808b6446b01c038a, entries=150, sequenceid=596, filesize=11.9 K 2023-04-03 02:55:47,391 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/4fafa47b2083432a872091def5cc0c60 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/4fafa47b2083432a872091def5cc0c60 2023-04-03 02:55:47,394 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/4fafa47b2083432a872091def5cc0c60, entries=150, sequenceid=596, filesize=11.9 K 2023-04-03 02:55:47,395 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=6.71 KB/6870 for 21fe738407d44fd93b895283a93c3a0c in 1645ms, sequenceid=596, compaction requested=false 2023-04-03 02:55:47,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:48,364 DEBUG [Thread-2464] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bb27fd4 to 127.0.0.1:52506 2023-04-03 02:55:48,364 DEBUG [Thread-2464] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:48,384 DEBUG [Thread-2466] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x445919fd to 127.0.0.1:52506 2023-04-03 02:55:48,384 DEBUG [Thread-2466] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:51,440 DEBUG [Thread-2468] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d010713 to 127.0.0.1:52506 2023-04-03 02:55:51,441 DEBUG [Thread-2468] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 146 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 135 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 97 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 68 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 81 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1707 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5118 rows 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1692 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5065 rows 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1679 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5033 rows 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1689 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5057 rows 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1696 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5088 rows 2023-04-03 02:55:51,441 INFO [Listener at localhost/46853] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 02:55:51,441 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6434ce29 to 127.0.0.1:52506 2023-04-03 02:55:51,442 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:55:51,443 INFO [Listener at localhost/46853] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 02:55:51,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 02:55:51,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:51,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 02:55:51,448 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490551448"}]},"ts":"1680490551448"} 2023-04-03 02:55:51,449 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 02:55:51,450 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 02:55:51,451 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=21fe738407d44fd93b895283a93c3a0c, UNASSIGN}] 2023-04-03 02:55:51,452 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=21fe738407d44fd93b895283a93c3a0c, UNASSIGN 2023-04-03 02:55:51,453 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=21fe738407d44fd93b895283a93c3a0c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:51,453 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490551453"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490551453"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490551453"}]},"ts":"1680490551453"} 2023-04-03 02:55:51,454 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:55:51,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 02:55:51,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:51,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 21fe738407d44fd93b895283a93c3a0c, disabling compactions & flushes 2023-04-03 02:55:51,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:51,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:51,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. after waiting 0 ms 2023-04-03 02:55:51,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:51,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 21fe738407d44fd93b895283a93c3a0c 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 02:55:51,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=A 2023-04-03 02:55:51,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:51,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=B 2023-04-03 02:55:51,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:51,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 21fe738407d44fd93b895283a93c3a0c, store=C 2023-04-03 02:55:51,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:51,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 02:55:52,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403ff2a4fbe4fda4606b3073f34af8cad5e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ff2a4fbe4fda4606b3073f34af8cad5e 2023-04-03 02:55:52,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 02:55:52,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=605, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/5c9757d2874d4899a60fbde2de89249f 2023-04-03 02:55:52,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 02:55:52,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/0937f4db7938433aa6763c6a6913cb93 2023-04-03 02:55:53,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/4cb3e8b6f839439988e3a9b7684e2f3b 2023-04-03 02:55:53,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/A/5c9757d2874d4899a60fbde2de89249f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/5c9757d2874d4899a60fbde2de89249f 2023-04-03 02:55:53,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/5c9757d2874d4899a60fbde2de89249f, entries=150, sequenceid=605, filesize=25.4 K 2023-04-03 02:55:53,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/B/0937f4db7938433aa6763c6a6913cb93 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/0937f4db7938433aa6763c6a6913cb93 2023-04-03 02:55:53,246 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/0937f4db7938433aa6763c6a6913cb93, entries=150, sequenceid=605, filesize=11.9 K 2023-04-03 02:55:53,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/.tmp/C/4cb3e8b6f839439988e3a9b7684e2f3b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/4cb3e8b6f839439988e3a9b7684e2f3b 2023-04-03 02:55:53,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/4cb3e8b6f839439988e3a9b7684e2f3b, entries=150, sequenceid=605, filesize=11.9 K 2023-04-03 02:55:53,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 21fe738407d44fd93b895283a93c3a0c in 1644ms, sequenceid=605, compaction requested=true 2023-04-03 02:55:53,250 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/712346ea572041ae96dca11f97363467, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/6ae410153c794f52afba27cac80cb265, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/02e47e13c6d2483da4b2b3986988c299, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/5c08514f7abe409fa6ed46d6943766f9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d42c2e1bf80349b48fc36917a079f617, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/eb4acb8b570b454aa2f3707f770b4b2c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c739137b4fba46a2ae780659643820ef, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/88fec2bc3e6b4e10bce6e700662432f7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d68791f129484e519d7c052d2f8cceb1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/9171ff41f1f6450fad05fd273be403ab] to archive 2023-04-03 02:55:53,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:55:53,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/712346ea572041ae96dca11f97363467 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/712346ea572041ae96dca11f97363467 2023-04-03 02:55:53,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/6ae410153c794f52afba27cac80cb265 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/6ae410153c794f52afba27cac80cb265 2023-04-03 02:55:53,254 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/02e47e13c6d2483da4b2b3986988c299 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/02e47e13c6d2483da4b2b3986988c299 2023-04-03 02:55:53,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/5c08514f7abe409fa6ed46d6943766f9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/5c08514f7abe409fa6ed46d6943766f9 2023-04-03 02:55:53,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d42c2e1bf80349b48fc36917a079f617 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d42c2e1bf80349b48fc36917a079f617 2023-04-03 02:55:53,256 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/eb4acb8b570b454aa2f3707f770b4b2c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/eb4acb8b570b454aa2f3707f770b4b2c 2023-04-03 02:55:53,257 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c739137b4fba46a2ae780659643820ef to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/c739137b4fba46a2ae780659643820ef 2023-04-03 02:55:53,258 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/88fec2bc3e6b4e10bce6e700662432f7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/88fec2bc3e6b4e10bce6e700662432f7 2023-04-03 02:55:53,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d68791f129484e519d7c052d2f8cceb1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/d68791f129484e519d7c052d2f8cceb1 2023-04-03 02:55:53,260 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/9171ff41f1f6450fad05fd273be403ab to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/9171ff41f1f6450fad05fd273be403ab 2023-04-03 02:55:53,261 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/e9748ea89e9a4f3aaeba07b388be36b9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/8103570fb9c744a6891a834f6f7b39cf, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/321a2ee7b9314f509ac35585ee3163fd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/983b90435fcf44ddacda198b23a7acc5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/d1fea2c012f24514944a43be373e94e0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16449f5c92ee424590e5c92de426db22, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/e2a79342f8fd4c09ab4c0fff1be0a73a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/ca0a09848b8e47dca7256573d105d2ea, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16fa9c0748594af495dcca8f41ff099b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/c41484f5105a4ec38fe474ca56b044cc] to archive 2023-04-03 02:55:53,262 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:55:53,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/e9748ea89e9a4f3aaeba07b388be36b9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/e9748ea89e9a4f3aaeba07b388be36b9 2023-04-03 02:55:53,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/8103570fb9c744a6891a834f6f7b39cf to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/8103570fb9c744a6891a834f6f7b39cf 2023-04-03 02:55:53,264 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/321a2ee7b9314f509ac35585ee3163fd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/321a2ee7b9314f509ac35585ee3163fd 2023-04-03 02:55:53,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/983b90435fcf44ddacda198b23a7acc5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/983b90435fcf44ddacda198b23a7acc5 2023-04-03 02:55:53,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/d1fea2c012f24514944a43be373e94e0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/d1fea2c012f24514944a43be373e94e0 2023-04-03 02:55:53,267 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16449f5c92ee424590e5c92de426db22 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16449f5c92ee424590e5c92de426db22 2023-04-03 02:55:53,267 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/e2a79342f8fd4c09ab4c0fff1be0a73a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/e2a79342f8fd4c09ab4c0fff1be0a73a 2023-04-03 02:55:53,268 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/ca0a09848b8e47dca7256573d105d2ea to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/ca0a09848b8e47dca7256573d105d2ea 2023-04-03 02:55:53,269 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16fa9c0748594af495dcca8f41ff099b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/16fa9c0748594af495dcca8f41ff099b 2023-04-03 02:55:53,269 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/c41484f5105a4ec38fe474ca56b044cc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/c41484f5105a4ec38fe474ca56b044cc 2023-04-03 02:55:53,271 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/17f8c565c0b243fc8cac69aee531c684, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/77c883f457994842b6eaf784838e7b3f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/46bcd1e136f34535b97caad0e1b04000, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/062d74701e25432fa37cc870d7f0b02d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/e4ac0fc6725e4f56904ff2a04dcd086a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ddb5c9cb08894194a5d84ed122dd399d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f5f1c603f8984342a17618b858a1f19c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/184ab9f6464140b78805d767dc6abfe1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f6955a89445145ce96c90b73170b9ca2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/154f33b1ec3a464e83441e687d004cbc] to archive 2023-04-03 02:55:53,271 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:55:53,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/17f8c565c0b243fc8cac69aee531c684 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/17f8c565c0b243fc8cac69aee531c684 2023-04-03 02:55:53,273 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/77c883f457994842b6eaf784838e7b3f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/77c883f457994842b6eaf784838e7b3f 2023-04-03 02:55:53,274 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/46bcd1e136f34535b97caad0e1b04000 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/46bcd1e136f34535b97caad0e1b04000 2023-04-03 02:55:53,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/062d74701e25432fa37cc870d7f0b02d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/062d74701e25432fa37cc870d7f0b02d 2023-04-03 02:55:53,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/e4ac0fc6725e4f56904ff2a04dcd086a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/e4ac0fc6725e4f56904ff2a04dcd086a 2023-04-03 02:55:53,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ddb5c9cb08894194a5d84ed122dd399d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/ddb5c9cb08894194a5d84ed122dd399d 2023-04-03 02:55:53,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f5f1c603f8984342a17618b858a1f19c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f5f1c603f8984342a17618b858a1f19c 2023-04-03 02:55:53,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/184ab9f6464140b78805d767dc6abfe1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/184ab9f6464140b78805d767dc6abfe1 2023-04-03 02:55:53,278 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f6955a89445145ce96c90b73170b9ca2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/f6955a89445145ce96c90b73170b9ca2 2023-04-03 02:55:53,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/154f33b1ec3a464e83441e687d004cbc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/154f33b1ec3a464e83441e687d004cbc 2023-04-03 02:55:53,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/recovered.edits/608.seqid, newMaxSeqId=608, maxSeqId=4 2023-04-03 02:55:53,283 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c. 2023-04-03 02:55:53,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 21fe738407d44fd93b895283a93c3a0c: 2023-04-03 02:55:53,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:53,284 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=21fe738407d44fd93b895283a93c3a0c, regionState=CLOSED 2023-04-03 02:55:53,285 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490553284"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490553284"}]},"ts":"1680490553284"} 2023-04-03 02:55:53,287 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-03 02:55:53,287 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 21fe738407d44fd93b895283a93c3a0c, server=jenkins-hbase4.apache.org,43815,1680490421348 in 1.8310 sec 2023-04-03 02:55:53,288 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-03 02:55:53,288 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=21fe738407d44fd93b895283a93c3a0c, UNASSIGN in 1.8360 sec 2023-04-03 02:55:53,288 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490553288"}]},"ts":"1680490553288"} 2023-04-03 02:55:53,289 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 02:55:53,290 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 02:55:53,291 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8470 sec 2023-04-03 02:55:53,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 02:55:53,551 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-03 02:55:53,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 02:55:53,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:53,554 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:53,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-03 02:55:53,554 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:53,559 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:53,560 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/recovered.edits] 2023-04-03 02:55:53,562 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/105b319be47c4b0697ff66f8bedc8af5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/105b319be47c4b0697ff66f8bedc8af5 2023-04-03 02:55:53,563 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/445d919ff0044c82b292cab9854f4033 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/445d919ff0044c82b292cab9854f4033 2023-04-03 02:55:53,565 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/5c9757d2874d4899a60fbde2de89249f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/A/5c9757d2874d4899a60fbde2de89249f 2023-04-03 02:55:53,567 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/0937f4db7938433aa6763c6a6913cb93 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/0937f4db7938433aa6763c6a6913cb93 2023-04-03 02:55:53,568 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/17f21c2702a54d10808b6446b01c038a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/17f21c2702a54d10808b6446b01c038a 2023-04-03 02:55:53,569 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/f545e3c7b9af4a5c97c7dfe6e22fb69a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/B/f545e3c7b9af4a5c97c7dfe6e22fb69a 2023-04-03 02:55:53,570 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/4cb3e8b6f839439988e3a9b7684e2f3b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/4cb3e8b6f839439988e3a9b7684e2f3b 2023-04-03 02:55:53,571 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/4fafa47b2083432a872091def5cc0c60 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/4fafa47b2083432a872091def5cc0c60 2023-04-03 02:55:53,572 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/cebefe00c6714caf84690bdae382ded0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/C/cebefe00c6714caf84690bdae382ded0 2023-04-03 02:55:53,574 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/recovered.edits/608.seqid to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c/recovered.edits/608.seqid 2023-04-03 02:55:53,575 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/21fe738407d44fd93b895283a93c3a0c 2023-04-03 02:55:53,575 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:55:53,575 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 02:55:53,575 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-03 02:55:53,577 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031751471f021c4663bfd5284520cf2873 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031751471f021c4663bfd5284520cf2873 2023-04-03 02:55:53,578 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031fb66df2fd5440bcbd940d3af13eaed2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031fb66df2fd5440bcbd940d3af13eaed2 2023-04-03 02:55:53,579 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033b5ec7f2bcf544188f832f65d46bf33e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033b5ec7f2bcf544188f832f65d46bf33e 2023-04-03 02:55:53,580 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034d1531a8499e4d309014976c5ded5c99 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034d1531a8499e4d309014976c5ded5c99 2023-04-03 02:55:53,581 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403522866c661e1409a9d0a859e6b31a022 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403522866c661e1409a9d0a859e6b31a022 2023-04-03 02:55:53,581 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035fac05cdf52d48fb8f85c9aabeb9e814 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035fac05cdf52d48fb8f85c9aabeb9e814 2023-04-03 02:55:53,582 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403636e583335f54a4db07b04419495d682 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403636e583335f54a4db07b04419495d682 2023-04-03 02:55:53,583 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403773f6b83c62f40098cfc281d820fd65e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403773f6b83c62f40098cfc281d820fd65e 2023-04-03 02:55:53,584 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040387f4e56678b5406fb980f93954fea929 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040387f4e56678b5406fb980f93954fea929 2023-04-03 02:55:53,585 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a04c1dd39b4d4b829dbe77ab4adb2f80 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a04c1dd39b4d4b829dbe77ab4adb2f80 2023-04-03 02:55:53,586 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a23ab04530e04d489d5ba6afedce789c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a23ab04530e04d489d5ba6afedce789c 2023-04-03 02:55:53,586 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a51bb73b894849cabf398f0aa9c702c6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a51bb73b894849cabf398f0aa9c702c6 2023-04-03 02:55:53,587 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ad212405e2494c2b9843fa9df7c37291 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ad212405e2494c2b9843fa9df7c37291 2023-04-03 02:55:53,588 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403cdef45d524b34a5baa7383199d11c592 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403cdef45d524b34a5baa7383199d11c592 2023-04-03 02:55:53,589 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d1a5ccf1254c40e394bf3e8d0003fb26 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d1a5ccf1254c40e394bf3e8d0003fb26 2023-04-03 02:55:53,590 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d22f813f0e0d4bdc98868461d1c222db to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d22f813f0e0d4bdc98868461d1c222db 2023-04-03 02:55:53,591 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f3f0d5a2619545b586febd168c4e9392 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f3f0d5a2619545b586febd168c4e9392 2023-04-03 02:55:53,592 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f619d7618e774e5ea35759568c6d9c63 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f619d7618e774e5ea35759568c6d9c63 2023-04-03 02:55:53,593 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403feb55fbacfa54e39a54c78d394bcdfd1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403feb55fbacfa54e39a54c78d394bcdfd1 2023-04-03 02:55:53,594 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ff2a4fbe4fda4606b3073f34af8cad5e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ff2a4fbe4fda4606b3073f34af8cad5e 2023-04-03 02:55:53,594 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 02:55:53,596 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:53,597 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 02:55:53,598 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 02:55:53,599 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:53,599 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 02:55:53,599 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680490553599"}]},"ts":"9223372036854775807"} 2023-04-03 02:55:53,600 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 02:55:53,600 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 21fe738407d44fd93b895283a93c3a0c, NAME => 'TestAcidGuarantees,,1680490520765.21fe738407d44fd93b895283a93c3a0c.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 02:55:53,601 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 02:55:53,601 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680490553601"}]},"ts":"9223372036854775807"} 2023-04-03 02:55:53,602 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 02:55:53,603 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:53,604 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 51 msec 2023-04-03 02:55:53,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-03 02:55:53,655 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-03 02:55:53,666 INFO [Listener at localhost/46853] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=226 (was 224) - Thread LEAK? -, OpenFileDescriptor=524 (was 513) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=440 (was 398) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 171), AvailableMemoryMB=3337 (was 3395) 2023-04-03 02:55:53,678 INFO [Listener at localhost/46853] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=226, OpenFileDescriptor=524, MaxFileDescriptor=60000, SystemLoadAverage=440, ProcessCount=170, AvailableMemoryMB=3337 2023-04-03 02:55:53,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 02:55:53,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 02:55:53,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 02:55:53,682 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 02:55:53,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-03 02:55:53,682 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 02:55:53,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 02:55:53,683 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:53,684 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5 empty. 2023-04-03 02:55:53,684 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:53,684 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:55:53,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 02:55:53,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 02:55:54,090 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 02:55:54,092 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9abe279a88a55aefb769568e98ff0ef5, NAME => 'TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp 2023-04-03 02:55:54,104 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:55:54,104 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9abe279a88a55aefb769568e98ff0ef5, disabling compactions & flushes 2023-04-03 02:55:54,104 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:54,104 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:54,104 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. after waiting 0 ms 2023-04-03 02:55:54,104 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:54,104 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:54,104 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:54,106 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 02:55:54,107 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490554107"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490554107"}]},"ts":"1680490554107"} 2023-04-03 02:55:54,108 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 02:55:54,109 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 02:55:54,109 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490554109"}]},"ts":"1680490554109"} 2023-04-03 02:55:54,109 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 02:55:54,113 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9abe279a88a55aefb769568e98ff0ef5, ASSIGN}] 2023-04-03 02:55:54,115 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9abe279a88a55aefb769568e98ff0ef5, ASSIGN 2023-04-03 02:55:54,115 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9abe279a88a55aefb769568e98ff0ef5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43815,1680490421348; forceNewPlan=false, retain=false 2023-04-03 02:55:54,266 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=9abe279a88a55aefb769568e98ff0ef5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,266 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490554266"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490554266"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490554266"}]},"ts":"1680490554266"} 2023-04-03 02:55:54,267 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:55:54,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 02:55:54,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:54,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9abe279a88a55aefb769568e98ff0ef5, NAME => 'TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:55:54,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:54,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:55:54,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:54,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:54,423 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:54,424 DEBUG [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A 2023-04-03 02:55:54,424 DEBUG [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A 2023-04-03 02:55:54,424 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:55:54,424 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9abe279a88a55aefb769568e98ff0ef5 columnFamilyName A 2023-04-03 02:55:54,425 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] regionserver.HStore(310): Store=9abe279a88a55aefb769568e98ff0ef5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:55:54,425 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:54,426 DEBUG [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B 2023-04-03 02:55:54,426 DEBUG [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B 2023-04-03 02:55:54,426 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:55:54,426 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9abe279a88a55aefb769568e98ff0ef5 columnFamilyName B 2023-04-03 02:55:54,426 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] regionserver.HStore(310): Store=9abe279a88a55aefb769568e98ff0ef5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:55:54,426 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:54,427 DEBUG [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C 2023-04-03 02:55:54,427 DEBUG [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C 2023-04-03 02:55:54,427 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:55:54,428 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9abe279a88a55aefb769568e98ff0ef5 columnFamilyName C 2023-04-03 02:55:54,428 INFO [StoreOpener-9abe279a88a55aefb769568e98ff0ef5-1] regionserver.HStore(310): Store=9abe279a88a55aefb769568e98ff0ef5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:55:54,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:54,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:54,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:54,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 02:55:54,431 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:54,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 02:55:54,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9abe279a88a55aefb769568e98ff0ef5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59652276, jitterRate=-0.11111181974411011}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 02:55:54,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:54,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., pid=49, masterSystemTime=1680490554419 2023-04-03 02:55:54,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:54,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:54,435 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=9abe279a88a55aefb769568e98ff0ef5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,435 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490554435"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680490554435"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680490554435"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680490554435"}]},"ts":"1680490554435"} 2023-04-03 02:55:54,437 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-03 02:55:54,437 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 in 169 msec 2023-04-03 02:55:54,438 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-03 02:55:54,438 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9abe279a88a55aefb769568e98ff0ef5, ASSIGN in 324 msec 2023-04-03 02:55:54,438 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 02:55:54,439 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490554438"}]},"ts":"1680490554438"} 2023-04-03 02:55:54,439 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 02:55:54,441 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 02:55:54,441 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 761 msec 2023-04-03 02:55:54,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 02:55:54,786 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-03 02:55:54,787 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x5e372785 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:54,792 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a987907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:54,793 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:54,795 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35250, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:54,796 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 02:55:54,798 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43774, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 02:55:54,799 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x501c5a88 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:54,804 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37705401, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:54,805 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x27f2d997 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:54,811 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15880ffb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:54,811 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x34ad4aa8 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:54,816 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52203cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:54,816 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x1c29757d to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:54,820 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420f945a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:54,820 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x06edd017 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:54,823 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51b8df2a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:54,824 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:54,828 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:54,829 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:54,834 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:54,835 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x5f5363d8 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:54,838 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78592df0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:54,838 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x211582f2 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:54,843 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@659c68ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:54,843 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x65816b6e to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:55:54,848 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@680e773e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:55:54,852 DEBUG [hconnection-0x75d8e8b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:54,852 DEBUG [hconnection-0x2ac1f732-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:54,852 DEBUG [hconnection-0x2198c7a6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:54,853 DEBUG [hconnection-0x1112132a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:54,853 DEBUG [hconnection-0x70c43878-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:54,854 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35258, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:54,854 DEBUG [hconnection-0x6e655efd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:54,856 DEBUG [hconnection-0x78e58414-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:54,856 DEBUG [hconnection-0x2cdd4b0f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:54,854 DEBUG [hconnection-0x71aa7480-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:54,855 DEBUG [hconnection-0x21e25c6a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:55:54,856 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35260, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:54,856 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35274, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:54,859 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35276, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:54,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:54,860 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35280, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:54,860 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:54,866 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35298, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:54,866 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35286, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:54,866 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35336, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:54,866 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:55:54,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:54,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:54,869 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:54,869 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:54,869 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:54,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:54,871 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:54,871 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:54,873 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,873 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:54,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:54,873 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,873 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:54,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:54,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:54,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:55:54,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:54,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:55:54,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:54,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:55:54,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:54,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:55:54,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:54,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:54,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:54,876 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:54,876 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:54,877 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:54,877 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:54,878 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:54,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:54,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:54,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:54,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ae32d2b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:54,881 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:54,881 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:54,882 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:54,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:54,882 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,882 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:54,884 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,884 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:54,884 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:54,884 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:54,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:54,887 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:54,887 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:54,887 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:54,887 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:54,887 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:54,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/32f84934fd0a4bd9a720c1ca4d2803cb 2023-04-03 02:55:54,889 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:54,889 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:54,889 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:54,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:54,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:54,891 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:54,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490554886, max timestamp=1680490554892Num uniques 859; 2023-04-03 02:55:54,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:54,894 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:54,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490554886, max timestamp=1680490554892Num uniques 859; 2023-04-03 02:55:54,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490554886, max timestamp=1680490554892Num uniques 859; 2023-04-03 02:55:54,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:54,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,896 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:54,896 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:54,896 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:54,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:54,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:54,896 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:54,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:54,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:54,896 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:54,897 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:54,897 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:54,897 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:54,897 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,897 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:54,897 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:54,897 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:54,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:54,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490614930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:54,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490614937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:54,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490614938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:54,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490614939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:54,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490614949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:54,997 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:54,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:54,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:55,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:55,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:55,001 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:55,001 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:55,001 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:55,001 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:55,002 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,002 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,003 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:55,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:55,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:55,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:55,013 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:55,013 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:55,013 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:55,013 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:55,015 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,023 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,023 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:55,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311dfc40[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:55,024 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:55,024 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,026 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,027 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:55,027 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:55,027 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,027 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:55,027 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:55,028 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,030 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:55,030 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:55,030 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:55,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:55,031 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:55,031 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:55,031 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:55,031 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:55,031 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:55,033 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,033 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:55,033 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:55,033 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,035 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490615040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490615041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,044 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:55,044 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:55,044 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:55,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:55,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:55,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:55,044 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:55,044 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,044 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,045 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,045 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,045 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:55,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:55,045 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:55,045 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,045 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490615050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490615050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490615055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,145 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:55,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:55,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:55,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:55,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:55,148 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:55,148 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:55,148 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:55,148 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:55,151 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,151 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:55,152 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:55,153 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,153 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:55,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:55,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:55,154 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:55,154 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:55,154 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:55,156 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:55,156 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:55,156 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:55,156 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:55,156 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:55,156 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:55,156 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:55,156 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:55,156 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,156 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:55,156 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:55,158 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,158 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,158 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,158 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:55,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8b141cc[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:55,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:55,159 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,162 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:55,162 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:55,162 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,162 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:55,163 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:55,163 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,163 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,163 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:55,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:55,166 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:55,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:55,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:55,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:55,166 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:55,167 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,167 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,167 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,168 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,169 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,171 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,171 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:55,171 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:55,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:55,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:55,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:55,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:55,171 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:55,171 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:55,171 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:55,171 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,172 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:55,172 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,172 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,172 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,172 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490615248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490615248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490615254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490615255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490615260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,272 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:55,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:55,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:55,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:55,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:55,275 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:55,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:55,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:55,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:55,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,278 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,278 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:55,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:55,278 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,278 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:55,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:55,279 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:55,279 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:55,280 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:55,280 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:55,280 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:55,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:55,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:55,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@266d37f3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:55,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,284 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:55,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:55,284 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:55,284 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,284 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:55,286 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:55,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:55,288 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:55,288 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:55,288 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:55,288 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:55,288 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:55,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:55,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:55,292 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,293 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,294 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,294 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:55,294 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:55,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:55,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:55,295 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,295 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,295 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,295 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,295 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:55,295 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:55,295 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:55,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:55,295 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:55,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/30182d05682641fd96bc74eadf0138cd 2023-04-03 02:55:55,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/b4169953bf7842c6bc437e7c16cde0bf 2023-04-03 02:55:55,318 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/32f84934fd0a4bd9a720c1ca4d2803cb as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/32f84934fd0a4bd9a720c1ca4d2803cb 2023-04-03 02:55:55,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/32f84934fd0a4bd9a720c1ca4d2803cb, entries=150, sequenceid=11, filesize=11.6 K 2023-04-03 02:55:55,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/30182d05682641fd96bc74eadf0138cd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/30182d05682641fd96bc74eadf0138cd 2023-04-03 02:55:55,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/30182d05682641fd96bc74eadf0138cd, entries=100, sequenceid=11, filesize=9.3 K 2023-04-03 02:55:55,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/b4169953bf7842c6bc437e7c16cde0bf as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b4169953bf7842c6bc437e7c16cde0bf 2023-04-03 02:55:55,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b4169953bf7842c6bc437e7c16cde0bf, entries=100, sequenceid=11, filesize=9.3 K 2023-04-03 02:55:55,328 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 9abe279a88a55aefb769568e98ff0ef5 in 453ms, sequenceid=11, compaction requested=false 2023-04-03 02:55:55,328 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:55,395 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:55,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:55,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:55,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:55,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:55,400 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:55,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:55,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:55,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:55,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,400 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,401 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,401 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:55,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:55,402 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,402 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:55,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:55,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:55,408 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:55,408 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:55,408 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:55,408 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:55,408 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-03 02:55:55,408 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:55:55,408 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:55,409 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:55:55,409 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:55,409 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:55:55,409 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:55,422 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/2434ab635f8045e292d93db2d16eca92 2023-04-03 02:55:55,442 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/332c3a8dfe564c1bb2a4aabd6f3ea666 2023-04-03 02:55:55,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:55,558 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:55,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490615565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490615565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490615566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490615567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490615567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490615668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490615671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490615671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490615673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,857 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/210366bd20cf44458b16f9977ab0a121 2023-04-03 02:55:55,861 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/2434ab635f8045e292d93db2d16eca92 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/2434ab635f8045e292d93db2d16eca92 2023-04-03 02:55:55,865 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/2434ab635f8045e292d93db2d16eca92, entries=150, sequenceid=48, filesize=11.6 K 2023-04-03 02:55:55,866 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/332c3a8dfe564c1bb2a4aabd6f3ea666 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/332c3a8dfe564c1bb2a4aabd6f3ea666 2023-04-03 02:55:55,869 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/332c3a8dfe564c1bb2a4aabd6f3ea666, entries=150, sequenceid=48, filesize=11.6 K 2023-04-03 02:55:55,870 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/210366bd20cf44458b16f9977ab0a121 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/210366bd20cf44458b16f9977ab0a121 2023-04-03 02:55:55,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,873 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/210366bd20cf44458b16f9977ab0a121, entries=150, sequenceid=48, filesize=11.6 K 2023-04-03 02:55:55,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490615871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,874 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 9abe279a88a55aefb769568e98ff0ef5 in 466ms, sequenceid=48, compaction requested=false 2023-04-03 02:55:55,874 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:55,874 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:55,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:55,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:55,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,876 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:55,876 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:55,877 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:55,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25e4dc60[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:55,880 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:55,880 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:55,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 02:55:55,882 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:55,882 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:55,882 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,882 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:55,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:55:55,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:55,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:55:55,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:55,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:55:55,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:55,883 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:55,883 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:55,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:55,890 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:55,890 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:55,890 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:55,890 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:55,891 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:55,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:55,892 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:55,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:55,893 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,895 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,898 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:55,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:55,898 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,898 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:55,898 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,898 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,898 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:55,898 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:55,898 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:55,898 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:55,898 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:55,900 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:55,900 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:55,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:55,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:55,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:55,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:55,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/e076428bec3d4feb98c861427706bee2 2023-04-03 02:55:55,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/9b596996541f47cfa7bc8077e6ca737f 2023-04-03 02:55:55,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/6ec0e6db198c41fe89165dffaec7aecb 2023-04-03 02:55:55,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/e076428bec3d4feb98c861427706bee2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e076428bec3d4feb98c861427706bee2 2023-04-03 02:55:55,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e076428bec3d4feb98c861427706bee2, entries=200, sequenceid=60, filesize=13.9 K 2023-04-03 02:55:55,954 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/9b596996541f47cfa7bc8077e6ca737f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/9b596996541f47cfa7bc8077e6ca737f 2023-04-03 02:55:55,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490615951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490615952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/9b596996541f47cfa7bc8077e6ca737f, entries=100, sequenceid=60, filesize=9.3 K 2023-04-03 02:55:55,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:55,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490615953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:55,959 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/6ec0e6db198c41fe89165dffaec7aecb as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/6ec0e6db198c41fe89165dffaec7aecb 2023-04-03 02:55:55,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/6ec0e6db198c41fe89165dffaec7aecb, entries=100, sequenceid=60, filesize=9.3 K 2023-04-03 02:55:55,963 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9abe279a88a55aefb769568e98ff0ef5 in 82ms, sequenceid=60, compaction requested=true 2023-04-03 02:55:55,963 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:55,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:55,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:55,963 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:55,963 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:55,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:55,964 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37990 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:55,964 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 30970 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:55,964 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/A is initiating minor compaction (all files) 2023-04-03 02:55:55,964 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/B is initiating minor compaction (all files) 2023-04-03 02:55:55,964 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/A in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:55,964 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/B in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:55,964 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/32f84934fd0a4bd9a720c1ca4d2803cb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/2434ab635f8045e292d93db2d16eca92, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e076428bec3d4feb98c861427706bee2] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=37.1 K 2023-04-03 02:55:55,964 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/30182d05682641fd96bc74eadf0138cd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/332c3a8dfe564c1bb2a4aabd6f3ea666, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/9b596996541f47cfa7bc8077e6ca737f] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=30.2 K 2023-04-03 02:55:55,965 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 32f84934fd0a4bd9a720c1ca4d2803cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680490554872 2023-04-03 02:55:55,965 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 30182d05682641fd96bc74eadf0138cd, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680490554873 2023-04-03 02:55:55,965 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 2434ab635f8045e292d93db2d16eca92, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680490554929 2023-04-03 02:55:55,965 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 332c3a8dfe564c1bb2a4aabd6f3ea666, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680490554929 2023-04-03 02:55:55,965 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e076428bec3d4feb98c861427706bee2, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490555565 2023-04-03 02:55:55,965 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 9b596996541f47cfa7bc8077e6ca737f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490555565 2023-04-03 02:55:55,974 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#A#compaction#419 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:55,981 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#B#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:55,988 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/84d5a088fbfd4d6e916beb9984407c0f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/84d5a088fbfd4d6e916beb9984407c0f 2023-04-03 02:55:55,992 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/A of 9abe279a88a55aefb769568e98ff0ef5 into 84d5a088fbfd4d6e916beb9984407c0f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:55,992 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:55,992 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/A, priority=13, startTime=1680490555963; duration=0sec 2023-04-03 02:55:55,992 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:55,992 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:55,993 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 30970 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:55,993 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/C is initiating minor compaction (all files) 2023-04-03 02:55:55,993 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/C in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:55,993 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b4169953bf7842c6bc437e7c16cde0bf, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/210366bd20cf44458b16f9977ab0a121, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/6ec0e6db198c41fe89165dffaec7aecb] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=30.2 K 2023-04-03 02:55:55,993 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting b4169953bf7842c6bc437e7c16cde0bf, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680490554873 2023-04-03 02:55:55,993 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 210366bd20cf44458b16f9977ab0a121, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680490554929 2023-04-03 02:55:55,994 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6ec0e6db198c41fe89165dffaec7aecb, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490555565 2023-04-03 02:55:55,998 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#C#compaction#421 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:55,998 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:55,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:56,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:56,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:56,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:56,004 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:56,004 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:56,004 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:56,004 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:56,004 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:56,004 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:56,009 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,009 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:56,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:56,010 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/a4b3f2e50a4042f2a9cff8d7674cd64d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a4b3f2e50a4042f2a9cff8d7674cd64d 2023-04-03 02:55:56,010 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,010 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:56,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:56,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:56,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:56,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:56,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:56,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:56,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:56,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:56,012 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:56,012 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:56,012 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:56,012 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 02:55:56,012 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:55:56,013 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:56,013 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:55:56,013 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:56,013 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:55:56,013 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:56,019 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/C of 9abe279a88a55aefb769568e98ff0ef5 into a4b3f2e50a4042f2a9cff8d7674cd64d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:56,020 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:56,020 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/C, priority=13, startTime=1680490555963; duration=0sec 2023-04-03 02:55:56,020 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:56,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:56,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:56,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490616072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490616074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490616075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490616076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490616174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490616178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490616181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490616183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490616382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490616386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,390 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/5a236f5134994231b940744b054ee034 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5a236f5134994231b940744b054ee034 2023-04-03 02:55:56,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490616389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,395 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/B of 9abe279a88a55aefb769568e98ff0ef5 into 5a236f5134994231b940744b054ee034(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:56,395 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:56,395 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/B, priority=13, startTime=1680490555963; duration=0sec 2023-04-03 02:55:56,395 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:56,423 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/69b9c779653947aca566a9f6ea7aca0b 2023-04-03 02:55:56,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490616683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490616690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490616691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:56,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490616696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:56,833 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/735326def0a34c3b8166a483bf79f95a 2023-04-03 02:55:57,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:57,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490617088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:57,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490617197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:57,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490617200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:57,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490617202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,241 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/419f83b0638c4bfd97affa2f780ddd0f 2023-04-03 02:55:57,245 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/69b9c779653947aca566a9f6ea7aca0b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/69b9c779653947aca566a9f6ea7aca0b 2023-04-03 02:55:57,249 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/69b9c779653947aca566a9f6ea7aca0b, entries=150, sequenceid=81, filesize=11.6 K 2023-04-03 02:55:57,253 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/735326def0a34c3b8166a483bf79f95a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/735326def0a34c3b8166a483bf79f95a 2023-04-03 02:55:57,257 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/735326def0a34c3b8166a483bf79f95a, entries=150, sequenceid=81, filesize=11.6 K 2023-04-03 02:55:57,257 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/419f83b0638c4bfd97affa2f780ddd0f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/419f83b0638c4bfd97affa2f780ddd0f 2023-04-03 02:55:57,260 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/419f83b0638c4bfd97affa2f780ddd0f, entries=150, sequenceid=81, filesize=11.6 K 2023-04-03 02:55:57,261 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9abe279a88a55aefb769568e98ff0ef5 in 1249ms, sequenceid=81, compaction requested=false 2023-04-03 02:55:57,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:57,261 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:57,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:57,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:57,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:57,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b45cb60[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:57,266 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:57,266 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,268 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,268 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,268 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:57,269 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:57,269 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:57,271 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:57,271 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:57,272 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:57,272 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:57,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:57,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:57,274 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:57,274 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:57,274 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:57,274 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:57,274 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:57,276 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,276 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:57,276 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:57,276 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,278 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,280 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,280 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:57,280 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:57,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:57,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:57,280 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:57,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:57,280 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:57,281 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:57,281 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:57,280 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:57,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,281 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,381 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:57,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:57,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:57,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:57,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:57,384 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:57,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:57,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:57,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:57,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,385 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,385 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:57,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:57,386 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,386 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:57,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:57,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:57,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:57,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:57,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:57,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:57,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:57,387 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:57,387 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:57,388 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:55:57,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:55:57,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:57,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:55:57,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:57,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:55:57,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:57,395 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/9065ae7439cd4742829a4e984f4765f1 2023-04-03 02:55:57,404 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/175cff5813124af790de887ea4808388 2023-04-03 02:55:57,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:57,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:57,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:57,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490617771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,812 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/2e81d33546b64baab3a6f50f4e70866e 2023-04-03 02:55:57,816 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/9065ae7439cd4742829a4e984f4765f1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/9065ae7439cd4742829a4e984f4765f1 2023-04-03 02:55:57,819 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/9065ae7439cd4742829a4e984f4765f1, entries=150, sequenceid=95, filesize=11.6 K 2023-04-03 02:55:57,819 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/175cff5813124af790de887ea4808388 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/175cff5813124af790de887ea4808388 2023-04-03 02:55:57,825 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/175cff5813124af790de887ea4808388, entries=150, sequenceid=95, filesize=11.6 K 2023-04-03 02:55:57,825 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/2e81d33546b64baab3a6f50f4e70866e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2e81d33546b64baab3a6f50f4e70866e 2023-04-03 02:55:57,829 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2e81d33546b64baab3a6f50f4e70866e, entries=150, sequenceid=95, filesize=11.6 K 2023-04-03 02:55:57,830 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9abe279a88a55aefb769568e98ff0ef5 in 442ms, sequenceid=95, compaction requested=true 2023-04-03 02:55:57,830 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:57,830 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:57,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:57,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:57,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,831 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:57,831 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,832 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:57,835 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:57,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79784ae5[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:57,835 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,836 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:57,837 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,837 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:57,838 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:57,838 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:57,838 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:57,839 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:57,842 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:57,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:57,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:57,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:57,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:57,842 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:57,847 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,847 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:57,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,847 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:57,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:57,849 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,851 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,853 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,853 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,853 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,853 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,853 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:57,854 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:57,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:57,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:57,853 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,854 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,854 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:57,854 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:57,854 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:57,854 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:57,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:57,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:57,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:57,883 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-03 02:55:57,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:55:57,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:57,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:55:57,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:57,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:55:57,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:57,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:57,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490617931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,954 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:57,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:57,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:57,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:57,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:57,957 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:57,957 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:57,957 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:57,957 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:57,960 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,960 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,961 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:57,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:57,961 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,963 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,963 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:57,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:57,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:57,965 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:57,965 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:57,965 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:57,965 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:57,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36653896[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:57,971 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:57,971 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,973 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,974 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:57,974 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:57,974 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:57,974 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,974 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:57,975 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,975 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:57,975 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:57,975 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:57,978 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:57,978 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:57,978 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:57,978 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:57,978 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:57,980 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:57,980 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,980 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:57,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:57,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:57,980 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,982 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,986 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,986 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:57,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:57,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:57,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:57,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:57,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:57,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:57,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:57,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:57,987 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:57,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:57,988 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:57,988 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:58,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:58,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490618041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,088 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:58,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:58,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:58,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:58,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:58,091 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:58,091 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:58,091 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:58,091 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:58,092 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,092 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,094 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,094 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:58,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:58,094 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,094 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:58,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:58,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:58,096 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:58,096 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:58,096 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:58,096 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:58,096 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:58,096 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:58,096 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:58,096 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:58,096 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:58,097 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:58,097 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:58,097 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,097 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:58,097 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:58,101 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,101 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:58,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:58,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35e1f5e6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:58,104 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:58,104 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,105 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:58,106 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,106 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:58,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:58,107 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:58,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:58,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:58,111 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:58,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:58,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:58,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:58,111 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:58,112 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,112 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,112 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:58,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:58,113 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,115 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:58,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:58,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:58,119 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:58,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:58,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:58,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:58,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:58,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:58,119 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:58,119 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:58,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:58,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490618207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:58,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490618210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:58,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490618216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,219 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:58,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:58,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:58,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:58,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:58,223 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:58,223 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:58,223 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:58,223 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:58,224 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,224 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,226 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,226 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:58,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:58,226 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,226 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:58,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:58,227 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:58,227 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:58,227 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:58,227 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:58,227 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:58,227 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:58,227 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:58,228 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:58,228 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:58,229 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:58,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:58,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@602a26f1[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:58,231 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:58,231 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,232 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,232 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:58,232 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:58,232 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:58,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:58,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:58,239 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:58,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:58,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:58,242 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:58,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:58,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:58,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:58,242 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:58,243 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,243 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:58,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,243 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:58,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:58,244 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,245 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,247 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:58,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:58,247 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:58,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:58,248 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,248 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,248 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,248 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:58,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:58,248 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:58,248 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:58,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:58,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:58,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490618245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/56ed001246b94ff789abefa79503a118 2023-04-03 02:55:58,317 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/75b87767ab8a43e3bb1fb9269ce8dbc9 2023-04-03 02:55:58,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/c109cc8794e34f32bf8b08c902cd7794 2023-04-03 02:55:58,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/56ed001246b94ff789abefa79503a118 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/56ed001246b94ff789abefa79503a118 2023-04-03 02:55:58,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/56ed001246b94ff789abefa79503a118, entries=200, sequenceid=116, filesize=13.9 K 2023-04-03 02:55:58,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/75b87767ab8a43e3bb1fb9269ce8dbc9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/75b87767ab8a43e3bb1fb9269ce8dbc9 2023-04-03 02:55:58,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/75b87767ab8a43e3bb1fb9269ce8dbc9, entries=150, sequenceid=116, filesize=11.6 K 2023-04-03 02:55:58,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/c109cc8794e34f32bf8b08c902cd7794 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c109cc8794e34f32bf8b08c902cd7794 2023-04-03 02:55:58,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c109cc8794e34f32bf8b08c902cd7794, entries=150, sequenceid=116, filesize=11.6 K 2023-04-03 02:55:58,341 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 9abe279a88a55aefb769568e98ff0ef5 in 458ms, sequenceid=116, compaction requested=true 2023-04-03 02:55:58,341 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:58,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:58,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:58,341 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:55:58,341 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:55:58,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:58,342 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49987 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:55:58,342 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/A is initiating minor compaction (all files) 2023-04-03 02:55:58,342 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/A in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:58,342 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:55:58,342 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/84d5a088fbfd4d6e916beb9984407c0f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/69b9c779653947aca566a9f6ea7aca0b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/9065ae7439cd4742829a4e984f4765f1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/56ed001246b94ff789abefa79503a118] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=48.8 K 2023-04-03 02:55:58,342 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/B is initiating minor compaction (all files) 2023-04-03 02:55:58,342 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/B in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:58,343 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5a236f5134994231b940744b054ee034, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/735326def0a34c3b8166a483bf79f95a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/175cff5813124af790de887ea4808388, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/75b87767ab8a43e3bb1fb9269ce8dbc9] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=46.5 K 2023-04-03 02:55:58,343 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 84d5a088fbfd4d6e916beb9984407c0f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490554939 2023-04-03 02:55:58,343 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 5a236f5134994231b940744b054ee034, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490554939 2023-04-03 02:55:58,343 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 69b9c779653947aca566a9f6ea7aca0b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1680490555950 2023-04-03 02:55:58,343 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 735326def0a34c3b8166a483bf79f95a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1680490555950 2023-04-03 02:55:58,343 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 9065ae7439cd4742829a4e984f4765f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680490556067 2023-04-03 02:55:58,344 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 175cff5813124af790de887ea4808388, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680490556067 2023-04-03 02:55:58,344 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 56ed001246b94ff789abefa79503a118, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680490557744 2023-04-03 02:55:58,344 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 75b87767ab8a43e3bb1fb9269ce8dbc9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680490557744 2023-04-03 02:55:58,348 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:58,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:58,350 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#A#compaction#431 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:58,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:58,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:58,352 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#B#compaction#432 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:58,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:58,352 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:58,352 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:58,352 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:58,352 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:58,353 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,353 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,354 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:58,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:58,354 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,354 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,354 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:58,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:58,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:58,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:58,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:58,361 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:58,361 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:58,361 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:58,362 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:58,362 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 02:55:58,362 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:55:58,362 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:58,362 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:55:58,362 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:58,362 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:55:58,362 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:58,371 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/defe6afc48b74d1e93aa0d9f5e528154 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/defe6afc48b74d1e93aa0d9f5e528154 2023-04-03 02:55:58,374 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/B of 9abe279a88a55aefb769568e98ff0ef5 into defe6afc48b74d1e93aa0d9f5e528154(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:58,375 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:58,375 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/B, priority=12, startTime=1680490558341; duration=0sec 2023-04-03 02:55:58,375 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:58,375 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:55:58,376 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:55:58,376 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/C is initiating minor compaction (all files) 2023-04-03 02:55:58,376 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/C in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:58,376 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a4b3f2e50a4042f2a9cff8d7674cd64d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/419f83b0638c4bfd97affa2f780ddd0f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2e81d33546b64baab3a6f50f4e70866e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c109cc8794e34f32bf8b08c902cd7794] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=46.5 K 2023-04-03 02:55:58,376 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a4b3f2e50a4042f2a9cff8d7674cd64d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680490554939 2023-04-03 02:55:58,377 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 419f83b0638c4bfd97affa2f780ddd0f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1680490555950 2023-04-03 02:55:58,377 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 2e81d33546b64baab3a6f50f4e70866e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680490556067 2023-04-03 02:55:58,377 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting c109cc8794e34f32bf8b08c902cd7794, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680490557744 2023-04-03 02:55:58,379 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/3eaf38e3c8f84c0ca445ebed6950b59e 2023-04-03 02:55:58,385 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#C#compaction#435 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:58,398 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/c81884513b7d475b99ed96fabbf5d4c5 2023-04-03 02:55:58,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:58,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:58,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490558619, max timestamp=1680490558670Num uniques 859; 2023-04-03 02:55:58,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:58,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490558619, max timestamp=1680490558670Num uniques 859; 2023-04-03 02:55:58,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490558619, max timestamp=1680490558670Num uniques 859; 2023-04-03 02:55:58,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:58,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:58,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490618746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,758 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/66e0cdcba4e04486b4fa01af298ec1a9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/66e0cdcba4e04486b4fa01af298ec1a9 2023-04-03 02:55:58,761 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/A of 9abe279a88a55aefb769568e98ff0ef5 into 66e0cdcba4e04486b4fa01af298ec1a9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:58,761 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:58,761 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/A, priority=12, startTime=1680490558341; duration=0sec 2023-04-03 02:55:58,762 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:58,802 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/65872f7177d748deaf6f4317e4a0e1c9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/65872f7177d748deaf6f4317e4a0e1c9 2023-04-03 02:55:58,806 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/C of 9abe279a88a55aefb769568e98ff0ef5 into 65872f7177d748deaf6f4317e4a0e1c9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:58,806 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:58,806 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/C, priority=12, startTime=1680490558341; duration=0sec 2023-04-03 02:55:58,806 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/e3112f2ff1834332955369b7d28391f1 2023-04-03 02:55:58,806 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:58,810 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/3eaf38e3c8f84c0ca445ebed6950b59e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3eaf38e3c8f84c0ca445ebed6950b59e 2023-04-03 02:55:58,815 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3eaf38e3c8f84c0ca445ebed6950b59e, entries=150, sequenceid=127, filesize=11.6 K 2023-04-03 02:55:58,815 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/c81884513b7d475b99ed96fabbf5d4c5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/c81884513b7d475b99ed96fabbf5d4c5 2023-04-03 02:55:58,819 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/c81884513b7d475b99ed96fabbf5d4c5, entries=150, sequenceid=127, filesize=11.6 K 2023-04-03 02:55:58,819 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/e3112f2ff1834332955369b7d28391f1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e3112f2ff1834332955369b7d28391f1 2023-04-03 02:55:58,822 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e3112f2ff1834332955369b7d28391f1, entries=150, sequenceid=127, filesize=11.6 K 2023-04-03 02:55:58,823 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9abe279a88a55aefb769568e98ff0ef5 in 461ms, sequenceid=127, compaction requested=false 2023-04-03 02:55:58,823 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:58,823 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:58,823 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:58,823 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:58,823 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:58,824 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 02:55:58,826 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bbe7773[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:58,828 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:58,828 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,829 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:58,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:58,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:58,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:58,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:58,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:58,832 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:58,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:58,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:58,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:58,832 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:58,834 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,834 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:58,834 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:58,836 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,837 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,840 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:58,840 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:58,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:58,840 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,840 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,840 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:58,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:58,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:58,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:58,842 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:58,842 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:58,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:58,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 02:55:58,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:55:58,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:58,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:55:58,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:58,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:55:58,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:55:58,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/d753a96a12ea43499c1a2cea7d65ea47 2023-04-03 02:55:58,877 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/904368f8af7749088daea9c265d2d6e0 2023-04-03 02:55:58,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:58,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490618894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,942 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:58,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:58,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:58,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:58,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:58,945 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:58,945 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:58,945 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:58,945 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:58,945 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,945 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,946 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,946 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:58,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:58,947 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,947 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:58,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:58,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:58,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:58,948 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:58,948 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:58,948 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:58,948 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:58,950 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,950 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:58,950 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,950 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4480551a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:58,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:58,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:58,953 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,953 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:58,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:58,960 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:58,960 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:58,960 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:58,961 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:58,961 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:58,962 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,962 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,962 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:58,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:58,962 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,963 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,965 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:58,965 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:58,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:58,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:58,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:58,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:58,966 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:58,966 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:58,966 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:58,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:58,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:58,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:58,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:58,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:59,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490619000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,066 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:59,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:59,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:59,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:59,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:59,079 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:59,079 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:59,079 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:59,080 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:59,080 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,080 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,081 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:59,081 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:59,082 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,082 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:59,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:59,086 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:59,089 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:59,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:59,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef4c6a9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:59,092 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:59,092 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,093 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:59,093 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,093 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:59,094 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:59,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:59,097 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:59,097 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:59,097 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:59,097 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:59,097 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:59,098 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,098 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:59,098 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:59,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,099 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:59,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:59,100 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:59,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,103 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:59,103 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:59,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:59,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:59,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:59,103 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,103 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:59,104 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,104 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,104 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,104 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,104 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,104 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,104 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:59,104 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:59,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:59,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:59,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:59,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490619102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,109 DEBUG [Thread-3082] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:55:59,204 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:59,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:59,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:59,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:59,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:59,207 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:59,207 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:59,207 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:59,207 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:59,208 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,208 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,209 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:59,209 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:59,210 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,210 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:59,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:59,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:59,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35276 deadline: 1680490619205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:59,211 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:59,211 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:59,211 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:59,211 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:55:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:55:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:55:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:55:59,213 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:59,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:59,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:59,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:55:59,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f586a28[Count = 0] remaining members to acquire global barrier 2023-04-03 02:55:59,215 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:55:59,215 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,216 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:55:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:55:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:55:59,216 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,216 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:55:59,220 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:55:59,220 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:59,220 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,220 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:55:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:55:59,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:55:59,222 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:55:59,222 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:55:59,222 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:55:59,222 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:55:59,222 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:55:59,224 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,224 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,224 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:59,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:55:59,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:59,224 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:55:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,225 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,227 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,227 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,227 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:55:59,227 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:59,228 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:55:59,227 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:59,228 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,228 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:55:59,228 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:55:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:55:59,228 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:55:59,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:55:59,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:55:59,228 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:55:59,228 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:55:59,285 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/07d110fe64ec428899ab55d1fc6d2243 2023-04-03 02:55:59,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/d753a96a12ea43499c1a2cea7d65ea47 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d753a96a12ea43499c1a2cea7d65ea47 2023-04-03 02:55:59,292 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d753a96a12ea43499c1a2cea7d65ea47, entries=200, sequenceid=167, filesize=14.1 K 2023-04-03 02:55:59,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/904368f8af7749088daea9c265d2d6e0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/904368f8af7749088daea9c265d2d6e0 2023-04-03 02:55:59,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/904368f8af7749088daea9c265d2d6e0, entries=150, sequenceid=167, filesize=11.8 K 2023-04-03 02:55:59,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/07d110fe64ec428899ab55d1fc6d2243 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/07d110fe64ec428899ab55d1fc6d2243 2023-04-03 02:55:59,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/07d110fe64ec428899ab55d1fc6d2243, entries=150, sequenceid=167, filesize=11.8 K 2023-04-03 02:55:59,299 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 9abe279a88a55aefb769568e98ff0ef5 in 441ms, sequenceid=167, compaction requested=true 2023-04-03 02:55:59,299 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:59,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:59,299 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:59,299 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:59,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:59,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:59,300 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38438 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:59,300 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:59,301 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/A is initiating minor compaction (all files) 2023-04-03 02:55:59,301 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/B is initiating minor compaction (all files) 2023-04-03 02:55:59,301 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/A in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:59,301 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/B in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:59,301 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/66e0cdcba4e04486b4fa01af298ec1a9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3eaf38e3c8f84c0ca445ebed6950b59e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d753a96a12ea43499c1a2cea7d65ea47] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=37.5 K 2023-04-03 02:55:59,301 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/defe6afc48b74d1e93aa0d9f5e528154, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/c81884513b7d475b99ed96fabbf5d4c5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/904368f8af7749088daea9c265d2d6e0] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=35.2 K 2023-04-03 02:55:59,301 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 66e0cdcba4e04486b4fa01af298ec1a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680490557744 2023-04-03 02:55:59,301 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting defe6afc48b74d1e93aa0d9f5e528154, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680490557744 2023-04-03 02:55:59,302 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting c81884513b7d475b99ed96fabbf5d4c5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680490557897 2023-04-03 02:55:59,302 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 3eaf38e3c8f84c0ca445ebed6950b59e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680490557897 2023-04-03 02:55:59,302 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 904368f8af7749088daea9c265d2d6e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680490558706 2023-04-03 02:55:59,302 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting d753a96a12ea43499c1a2cea7d65ea47, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680490558706 2023-04-03 02:55:59,310 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#A#compaction#440 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:59,311 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#B#compaction#441 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:59,329 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:55:59,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:55:59,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:55:59,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:55:59,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:55:59,332 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:55:59,332 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:59,332 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:55:59,332 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:55:59,333 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,333 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:55:59,334 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:55:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:55:59,334 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,334 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:55:59,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:55:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:55:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:55:59,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:55:59,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:55:59,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:55:59,335 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:55:59,335 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:55:59,335 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:55:59,336 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:55:59,336 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:55:59,336 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:59,336 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:55:59,336 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:55:59,336 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:55:59,336 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:59,336 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:55:59,336 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:59,336 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:55:59,336 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:55:59,348 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/95d8a99637714e2c9e561368ee4c518c 2023-04-03 02:55:59,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:55:59,518 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:55:59,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490559615, max timestamp=1680490559644Num uniques 859; 2023-04-03 02:55:59,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:59,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490559615, max timestamp=1680490559644Num uniques 859; 2023-04-03 02:55:59,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490559615, max timestamp=1680490559644Num uniques 859; 2023-04-03 02:55:59,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:59,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:55:59,730 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/9badc5da1e034e80898e6f6d5090966f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/9badc5da1e034e80898e6f6d5090966f 2023-04-03 02:55:59,732 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/a60db7e6f3174ba8bbadedf9d0dfab90 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/a60db7e6f3174ba8bbadedf9d0dfab90 2023-04-03 02:55:59,737 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/B of 9abe279a88a55aefb769568e98ff0ef5 into 9badc5da1e034e80898e6f6d5090966f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:59,738 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:59,738 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/B, priority=13, startTime=1680490559299; duration=0sec 2023-04-03 02:55:59,738 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:55:59,738 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:55:59,738 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/A of 9abe279a88a55aefb769568e98ff0ef5 into a60db7e6f3174ba8bbadedf9d0dfab90(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:59,738 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:59,738 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/A, priority=13, startTime=1680490559299; duration=0sec 2023-04-03 02:55:59,738 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:59,739 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:55:59,739 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/C is initiating minor compaction (all files) 2023-04-03 02:55:59,739 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/C in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:55:59,739 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/65872f7177d748deaf6f4317e4a0e1c9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e3112f2ff1834332955369b7d28391f1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/07d110fe64ec428899ab55d1fc6d2243] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=35.2 K 2023-04-03 02:55:59,739 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 65872f7177d748deaf6f4317e4a0e1c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680490557744 2023-04-03 02:55:59,739 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e3112f2ff1834332955369b7d28391f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680490557897 2023-04-03 02:55:59,740 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 07d110fe64ec428899ab55d1fc6d2243, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680490558706 2023-04-03 02:55:59,746 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#C#compaction#444 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:55:59,753 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/8a10170a9b804e148f24b82f44accb79 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/8a10170a9b804e148f24b82f44accb79 2023-04-03 02:55:59,756 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/C of 9abe279a88a55aefb769568e98ff0ef5 into 8a10170a9b804e148f24b82f44accb79(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:55:59,756 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:55:59,756 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/C, priority=13, startTime=1680490559300; duration=0sec 2023-04-03 02:55:59,756 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:55:59,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:59,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490619754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:55:59,757 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/b901c8b84b854acc88301279effbb54d 2023-04-03 02:55:59,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:55:59,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490619858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490620064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,167 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/d309d8f3a1bf44de91bfcfc6521067a8 2023-04-03 02:56:00,170 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/95d8a99637714e2c9e561368ee4c518c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/95d8a99637714e2c9e561368ee4c518c 2023-04-03 02:56:00,173 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/95d8a99637714e2c9e561368ee4c518c, entries=150, sequenceid=177, filesize=11.8 K 2023-04-03 02:56:00,174 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/b901c8b84b854acc88301279effbb54d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b901c8b84b854acc88301279effbb54d 2023-04-03 02:56:00,177 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b901c8b84b854acc88301279effbb54d, entries=150, sequenceid=177, filesize=11.8 K 2023-04-03 02:56:00,178 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/d309d8f3a1bf44de91bfcfc6521067a8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d309d8f3a1bf44de91bfcfc6521067a8 2023-04-03 02:56:00,181 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d309d8f3a1bf44de91bfcfc6521067a8, entries=150, sequenceid=177, filesize=11.8 K 2023-04-03 02:56:00,181 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9abe279a88a55aefb769568e98ff0ef5 in 845ms, sequenceid=177, compaction requested=false 2023-04-03 02:56:00,182 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:00,182 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:00,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:00,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:00,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:00,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:00,184 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:00,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67f5895f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:00,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:00,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,187 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:00,187 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,188 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:00,189 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,189 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:00,189 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,189 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:00,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:00,191 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:00,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:00,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:00,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:00,191 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:00,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,196 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,197 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,198 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:00,200 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:00,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:00,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:00,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:00,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:00,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:00,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,200 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:00,201 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:00,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:00,218 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-03 02:56:00,219 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:00,219 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 02:56:00,219 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:00,219 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 02:56:00,219 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:00,219 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 02:56:00,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490620234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490620235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490620238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,301 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:00,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:00,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:00,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:00,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:00,304 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:00,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:00,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:00,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:00,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,306 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:00,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:00,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:00,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:00,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:00,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:00,308 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:00,308 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:00,308 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:00,308 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:00,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,310 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:00,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@211b383e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:00,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:00,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,313 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:00,314 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,314 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:00,315 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:00,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:00,318 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:00,318 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:00,318 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:00,318 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:00,318 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:00,320 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,320 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:00,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,320 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:00,320 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,322 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,323 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,323 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,323 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,324 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:00,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:00,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:00,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:00,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:00,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:00,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:00,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:00,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,324 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:00,324 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:00,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490620339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490620342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490620345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490620369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,424 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:00,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:00,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:00,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:00,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:00,427 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:00,427 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:00,427 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:00,427 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:00,428 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,428 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,430 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:00,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:00,430 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:00,430 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,430 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:00,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:00,431 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:00,431 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:00,431 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:00,431 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:00,431 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:00,431 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:00,431 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:00,432 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:00,432 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:00,433 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:00,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7982733a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:00,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:00,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,436 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:00,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:00,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:00,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:00,437 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:00,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:00,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:00,439 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:00,439 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:00,439 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:00,440 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:00,440 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:00,442 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,442 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,442 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:00,443 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,445 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,447 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:00,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:00,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:00,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:00,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:00,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:00,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:00,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,447 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:00,448 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:00,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:00,548 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:00,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:00,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490620546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:00,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:00,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:00,551 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:00,551 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:00,551 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:00,551 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:00,551 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,551 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,552 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:00,552 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:00,553 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,553 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:00,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:00,554 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:00,554 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:00,554 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:00,554 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:00,554 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:00,554 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:00,554 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:00,554 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:00,554 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,556 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,556 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:00,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490620550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75075f08[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:00,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:00,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,560 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:00,560 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,560 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:00,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490620556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:00,565 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:00,565 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:00,565 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:00,565 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:00,565 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:00,566 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,566 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:00,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,566 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:00,567 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,568 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:00,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:00,574 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:00,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:00,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:00,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:00,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,574 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:00,574 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:00,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:00,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:00,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/933647ad9cea494b8a684ad4dbcca84c 2023-04-03 02:56:00,635 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/ec4449ccfb94441fa9e2982ef3794a98 2023-04-03 02:56:00,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/054daed70ff044c299c47387203ac7d4 2023-04-03 02:56:00,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/933647ad9cea494b8a684ad4dbcca84c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/933647ad9cea494b8a684ad4dbcca84c 2023-04-03 02:56:00,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/933647ad9cea494b8a684ad4dbcca84c, entries=200, sequenceid=218, filesize=14.1 K 2023-04-03 02:56:00,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/ec4449ccfb94441fa9e2982ef3794a98 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4449ccfb94441fa9e2982ef3794a98 2023-04-03 02:56:00,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4449ccfb94441fa9e2982ef3794a98, entries=150, sequenceid=218, filesize=11.8 K 2023-04-03 02:56:00,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/054daed70ff044c299c47387203ac7d4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/054daed70ff044c299c47387203ac7d4 2023-04-03 02:56:00,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/054daed70ff044c299c47387203ac7d4, entries=150, sequenceid=218, filesize=11.8 K 2023-04-03 02:56:00,657 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 9abe279a88a55aefb769568e98ff0ef5 in 439ms, sequenceid=218, compaction requested=true 2023-04-03 02:56:00,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:00,658 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:00,658 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:00,658 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:00,658 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:00,658 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:00,658 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38840 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:00,659 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/A is initiating minor compaction (all files) 2023-04-03 02:56:00,659 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/A in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:00,659 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/a60db7e6f3174ba8bbadedf9d0dfab90, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/95d8a99637714e2c9e561368ee4c518c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/933647ad9cea494b8a684ad4dbcca84c] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=37.9 K 2023-04-03 02:56:00,659 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:00,659 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/B is initiating minor compaction (all files) 2023-04-03 02:56:00,659 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/B in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:00,659 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a60db7e6f3174ba8bbadedf9d0dfab90, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680490558706 2023-04-03 02:56:00,659 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/9badc5da1e034e80898e6f6d5090966f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b901c8b84b854acc88301279effbb54d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4449ccfb94441fa9e2982ef3794a98] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=35.6 K 2023-04-03 02:56:00,659 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 95d8a99637714e2c9e561368ee4c518c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1680490558880 2023-04-03 02:56:00,660 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 933647ad9cea494b8a684ad4dbcca84c, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680490559709 2023-04-03 02:56:00,660 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 9badc5da1e034e80898e6f6d5090966f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680490558706 2023-04-03 02:56:00,661 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b901c8b84b854acc88301279effbb54d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1680490558880 2023-04-03 02:56:00,661 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ec4449ccfb94441fa9e2982ef3794a98, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680490559709 2023-04-03 02:56:00,666 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#A#compaction#449 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:00,667 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#B#compaction#450 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:00,674 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/ddc2eb59d5114ec9ad16cd393cb97851 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ddc2eb59d5114ec9ad16cd393cb97851 2023-04-03 02:56:00,674 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:00,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:00,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:00,677 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/B of 9abe279a88a55aefb769568e98ff0ef5 into ddc2eb59d5114ec9ad16cd393cb97851(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:00,677 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:00,677 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/B, priority=13, startTime=1680490560658; duration=0sec 2023-04-03 02:56:00,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:00,677 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:00,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:00,677 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:00,678 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:00,678 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:00,678 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:00,678 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:00,678 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,678 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,678 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:00,678 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/C is initiating minor compaction (all files) 2023-04-03 02:56:00,678 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/C in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:00,679 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/8a10170a9b804e148f24b82f44accb79, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d309d8f3a1bf44de91bfcfc6521067a8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/054daed70ff044c299c47387203ac7d4] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=35.6 K 2023-04-03 02:56:00,679 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 8a10170a9b804e148f24b82f44accb79, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680490558706 2023-04-03 02:56:00,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:00,679 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:00,679 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d309d8f3a1bf44de91bfcfc6521067a8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1680490558880 2023-04-03 02:56:00,679 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,679 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,680 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 054daed70ff044c299c47387203ac7d4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680490559709 2023-04-03 02:56:00,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:00,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:00,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:00,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:00,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:00,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:00,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:00,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:00,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:00,681 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:00,681 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:00,682 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-03 02:56:00,682 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:00,682 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:00,682 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:00,682 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:00,682 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:00,682 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:00,686 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#C#compaction#452 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:00,689 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/11a658354a17442ca56355e54e932d11 2023-04-03 02:56:00,698 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/c8b01a2261ec4c2b879240b5b713d3bd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c8b01a2261ec4c2b879240b5b713d3bd 2023-04-03 02:56:00,701 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/C of 9abe279a88a55aefb769568e98ff0ef5 into c8b01a2261ec4c2b879240b5b713d3bd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:00,701 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:00,702 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/C, priority=13, startTime=1680490560658; duration=0sec 2023-04-03 02:56:00,702 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:00,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:00,853 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490560864, max timestamp=1680490560888Num uniques 859; 2023-04-03 02:56:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490560864, max timestamp=1680490560888Num uniques 859; 2023-04-03 02:56:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490560864, max timestamp=1680490560888Num uniques 859; 2023-04-03 02:56:00,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:00,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490620909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490620913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490620914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:00,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:00,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490620914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490621016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490621016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490621020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490621020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,075 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/4e7245ae302c4b3a9b0b2cf121155381 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/4e7245ae302c4b3a9b0b2cf121155381 2023-04-03 02:56:01,079 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/A of 9abe279a88a55aefb769568e98ff0ef5 into 4e7245ae302c4b3a9b0b2cf121155381(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:01,079 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:01,079 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/A, priority=13, startTime=1680490560658; duration=0sec 2023-04-03 02:56:01,080 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:01,106 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/1f4eca250cb042049440700d7943cd34 2023-04-03 02:56:01,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490621223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490621223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490621225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490621226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,516 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/d06bfbe8bb054e62b0edf093fba3aeb1 2023-04-03 02:56:01,519 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/11a658354a17442ca56355e54e932d11 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/11a658354a17442ca56355e54e932d11 2023-04-03 02:56:01,522 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/11a658354a17442ca56355e54e932d11, entries=150, sequenceid=228, filesize=11.8 K 2023-04-03 02:56:01,523 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/1f4eca250cb042049440700d7943cd34 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1f4eca250cb042049440700d7943cd34 2023-04-03 02:56:01,526 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1f4eca250cb042049440700d7943cd34, entries=150, sequenceid=228, filesize=11.8 K 2023-04-03 02:56:01,526 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/d06bfbe8bb054e62b0edf093fba3aeb1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d06bfbe8bb054e62b0edf093fba3aeb1 2023-04-03 02:56:01,529 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d06bfbe8bb054e62b0edf093fba3aeb1, entries=150, sequenceid=228, filesize=11.8 K 2023-04-03 02:56:01,530 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 9abe279a88a55aefb769568e98ff0ef5 in 849ms, sequenceid=228, compaction requested=false 2023-04-03 02:56:01,530 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:01,530 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:01,532 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:01,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ef38f86[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:01,534 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:01,534 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,535 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,535 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,535 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:01,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490560912, max timestamp=1680490561533Num uniques 1068; 2023-04-03 02:56:01,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-03 02:56:01,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490560912, max timestamp=1680490561533Num uniques 1068; 2023-04-03 02:56:01,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-03 02:56:01,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:01,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=67.09 KB heapSize=183.88 KB 2023-04-03 02:56:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:01,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:01,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:01,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:01,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:01,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:01,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 02:56:01,538 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:01,538 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:01,538 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,538 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:01,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:01,541 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:01,541 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:01,541 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:01,541 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:01,541 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:01,542 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,542 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,542 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,543 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/e20eb0cbf6e14055a4d69e0b90b239e0 2023-04-03 02:56:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,545 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:01,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:01,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:01,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:01,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:01,547 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:01,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,548 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,548 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,547 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:01,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:01,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:01,548 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:01,548 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,548 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490621558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490621559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490621560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490621566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,648 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:01,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:01,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:01,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:01,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:01,651 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:01,651 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:01,651 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:01,651 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:01,651 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,651 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,653 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:01,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:01,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:01,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:01,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:01,654 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:01,654 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:01,654 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:01,654 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:01,654 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:01,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:01,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:01,660 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,660 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,660 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,660 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d18aae[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:01,661 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:01,661 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,662 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,663 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,663 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:01,663 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:01,663 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:01,663 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:01,664 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:01,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:01,666 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:01,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:01,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:01,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:01,667 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:01,668 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,668 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,668 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:01,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490621667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,673 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:01,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:01,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490621667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,673 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:01,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:01,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490621668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,674 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:01,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490621668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,674 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:01,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,674 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:01,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:01,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:01,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:01,774 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:01,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:01,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:01,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:01,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:01,778 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:01,778 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:01,778 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:01,778 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:01,778 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,778 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,779 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:01,779 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:01,780 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,780 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:01,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:01,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:01,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:01,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:01,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:01,782 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:01,782 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:01,782 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:01,782 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,782 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:01,782 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:01,783 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,784 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ca03aed[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:01,785 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:01,785 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,786 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,786 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:01,786 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:01,787 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:01,787 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,787 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:01,789 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:01,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:01,791 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:01,791 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:01,791 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:01,791 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:01,791 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:01,793 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,793 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,793 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,795 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,797 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:01,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,798 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:01,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:01,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:01,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:01,798 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:01,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:01,798 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:01,798 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:01,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:01,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490621874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490621875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490621875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:01,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490621876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,898 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:01,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:01,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:01,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:01,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:01,903 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:01,903 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:01,903 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:01,903 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:01,904 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,904 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,909 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,909 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,909 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:01,916 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:01,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:01,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:01,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:01,925 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:01,925 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:01,925 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:01,925 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:01,927 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,927 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,928 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6390df6e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:01,932 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:01,932 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,934 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,934 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:01,934 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:01,934 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:01,935 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,935 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:01,936 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,936 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:01,937 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:01,937 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:01,941 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:01,941 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:01,941 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:01,941 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:01,941 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:01,944 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,944 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,946 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:01,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,947 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,949 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,951 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,951 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:01,951 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:01,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:01,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:01,952 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:01,952 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:01,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:01,952 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:01,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:01,952 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:01,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:01,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:01,953 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:01,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/5991b3b8877c47f2b920166fffe25b27 2023-04-03 02:56:01,972 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/476aaf5a7a684c2ca4aeeefecd6f0158 2023-04-03 02:56:01,976 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/e20eb0cbf6e14055a4d69e0b90b239e0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e20eb0cbf6e14055a4d69e0b90b239e0 2023-04-03 02:56:01,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e20eb0cbf6e14055a4d69e0b90b239e0, entries=250, sequenceid=269, filesize=16.7 K 2023-04-03 02:56:01,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/5991b3b8877c47f2b920166fffe25b27 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5991b3b8877c47f2b920166fffe25b27 2023-04-03 02:56:01,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5991b3b8877c47f2b920166fffe25b27, entries=150, sequenceid=269, filesize=11.9 K 2023-04-03 02:56:01,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/476aaf5a7a684c2ca4aeeefecd6f0158 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/476aaf5a7a684c2ca4aeeefecd6f0158 2023-04-03 02:56:01,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/476aaf5a7a684c2ca4aeeefecd6f0158, entries=150, sequenceid=269, filesize=11.9 K 2023-04-03 02:56:01,992 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~203.75 KB/208640, currentSize=107.34 KB/109920 for 9abe279a88a55aefb769568e98ff0ef5 in 456ms, sequenceid=269, compaction requested=true 2023-04-03 02:56:01,992 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:01,993 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:01,993 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:01,993 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:01,993 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:01,993 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:01,994 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41582 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:01,994 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:01,994 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/A is initiating minor compaction (all files) 2023-04-03 02:56:01,994 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/B is initiating minor compaction (all files) 2023-04-03 02:56:01,994 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/A in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:01,994 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/B in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:01,994 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/4e7245ae302c4b3a9b0b2cf121155381, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/11a658354a17442ca56355e54e932d11, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e20eb0cbf6e14055a4d69e0b90b239e0] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=40.6 K 2023-04-03 02:56:01,994 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ddc2eb59d5114ec9ad16cd393cb97851, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1f4eca250cb042049440700d7943cd34, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5991b3b8877c47f2b920166fffe25b27] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=35.8 K 2023-04-03 02:56:01,994 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 4e7245ae302c4b3a9b0b2cf121155381, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680490559709 2023-04-03 02:56:01,994 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ddc2eb59d5114ec9ad16cd393cb97851, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680490559709 2023-04-03 02:56:01,994 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 11a658354a17442ca56355e54e932d11, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680490560233 2023-04-03 02:56:01,994 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 1f4eca250cb042049440700d7943cd34, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680490560233 2023-04-03 02:56:01,995 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e20eb0cbf6e14055a4d69e0b90b239e0, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680490560912 2023-04-03 02:56:01,995 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 5991b3b8877c47f2b920166fffe25b27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680490560913 2023-04-03 02:56:02,001 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#B#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:02,001 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#A#compaction#458 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:02,053 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:02,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:02,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:02,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:02,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:02,056 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:02,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:02,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:02,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,057 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:02,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:02,058 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,058 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:02,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:02,060 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:02,060 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,060 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:02,060 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 02:56:02,060 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:02,060 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:02,060 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:02,060 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:02,060 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:02,060 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:02,082 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/ce2cc877b43a4e03a788d88dd73f463e 2023-04-03 02:56:02,092 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/8ab35e192f4148c3a0fe153c1e503640 2023-04-03 02:56:02,105 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/f0057dfde4164aa5966be242656f0472 2023-04-03 02:56:02,108 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/ce2cc877b43a4e03a788d88dd73f463e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ce2cc877b43a4e03a788d88dd73f463e 2023-04-03 02:56:02,112 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ce2cc877b43a4e03a788d88dd73f463e, entries=150, sequenceid=288, filesize=11.9 K 2023-04-03 02:56:02,113 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/8ab35e192f4148c3a0fe153c1e503640 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8ab35e192f4148c3a0fe153c1e503640 2023-04-03 02:56:02,118 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8ab35e192f4148c3a0fe153c1e503640, entries=150, sequenceid=288, filesize=11.9 K 2023-04-03 02:56:02,119 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/f0057dfde4164aa5966be242656f0472 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/f0057dfde4164aa5966be242656f0472 2023-04-03 02:56:02,129 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/f0057dfde4164aa5966be242656f0472, entries=150, sequenceid=288, filesize=11.9 K 2023-04-03 02:56:02,131 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 9abe279a88a55aefb769568e98ff0ef5 in 71ms, sequenceid=288, compaction requested=true 2023-04-03 02:56:02,131 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:02,131 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:02,133 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,133 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:02,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413df325[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:02,135 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:02,135 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:02,139 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:02,139 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:02,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,139 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:02,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:02,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:02,139 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:02,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:02,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:02,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:02,140 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:02,141 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,142 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,145 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,147 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:02,147 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:02,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:02,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:02,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:02,147 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:02,147 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,148 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,148 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,148 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,148 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,148 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,148 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:02,148 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:02,148 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:02,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:02,195 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:56:02,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:02,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:02,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:02,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:02,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:02,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:02,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/8f87704b24ba4e34a8ee8abf920a939b 2023-04-03 02:56:02,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/1a7dea6b390542f8a8e9e38d547a4945 2023-04-03 02:56:02,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/7dca4342f8924b849c3355691c4f3c02 2023-04-03 02:56:02,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490562232, max timestamp=1680490562244Num uniques 859; 2023-04-03 02:56:02,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490562232, max timestamp=1680490562244Num uniques 859; 2023-04-03 02:56:02,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:02,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:02,248 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:02,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490562232, max timestamp=1680490562244Num uniques 859; 2023-04-03 02:56:02,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:02,248 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/8f87704b24ba4e34a8ee8abf920a939b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8f87704b24ba4e34a8ee8abf920a939b 2023-04-03 02:56:02,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:02,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:02,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:02,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:02,253 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:02,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,253 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8f87704b24ba4e34a8ee8abf920a939b, entries=250, sequenceid=299, filesize=16.7 K 2023-04-03 02:56:02,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:02,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:02,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,254 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,254 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/1a7dea6b390542f8a8e9e38d547a4945 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1a7dea6b390542f8a8e9e38d547a4945 2023-04-03 02:56:02,255 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:02,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:02,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:02,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:02,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:02,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:02,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:02,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:02,258 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1a7dea6b390542f8a8e9e38d547a4945, entries=150, sequenceid=299, filesize=11.9 K 2023-04-03 02:56:02,258 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,258 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:02,259 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:02,259 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:02,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:02,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:02,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:02,261 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/7dca4342f8924b849c3355691c4f3c02 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7dca4342f8924b849c3355691c4f3c02 2023-04-03 02:56:02,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:02,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b739556[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:02,263 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:02,263 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:02,265 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:02,265 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,265 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:02,266 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:02,266 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:02,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,266 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7dca4342f8924b849c3355691c4f3c02, entries=150, sequenceid=299, filesize=11.9 K 2023-04-03 02:56:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,268 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=73.80 KB/75570 for 9abe279a88a55aefb769568e98ff0ef5 in 73ms, sequenceid=299, compaction requested=true 2023-04-03 02:56:02,268 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,268 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:56:02,268 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-03 02:56:02,268 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-03 02:56:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:02,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:02,269 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:02,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:02,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:02,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:02,269 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:02,272 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,272 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,272 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:02,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,273 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,273 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=80.51 KB heapSize=220.92 KB 2023-04-03 02:56:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:02,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:02,276 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:02,276 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:02,276 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:02,276 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:02,279 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:02,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:02,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:02,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:02,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:02,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:02,285 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:02,286 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:02,286 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:02,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:02,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490622310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490622311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490622315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490622316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,386 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:02,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:02,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:02,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:02,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:02,389 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:02,389 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,389 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:02,389 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:02,389 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,391 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:02,391 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:02,392 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,392 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:02,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:02,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:02,394 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,394 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:02,394 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:02,394 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:02,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:02,399 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,400 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:02,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2629b187[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:02,401 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:02,401 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,402 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:02,402 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,402 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:02,404 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:02,404 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:02,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:02,406 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:02,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:02,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:02,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:02,406 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:02,408 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,408 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,408 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:02,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,409 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,410 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,413 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:02,413 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,413 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:02,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:02,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:02,413 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:02,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:02,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:02,413 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,414 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,414 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,414 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,414 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,414 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:02,414 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:02,414 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:02,417 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/80894bd081374f12828ba6e3440a26b8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/80894bd081374f12828ba6e3440a26b8 2023-04-03 02:56:02,420 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/B of 9abe279a88a55aefb769568e98ff0ef5 into 80894bd081374f12828ba6e3440a26b8(size=12.4 K), total size for store is 36.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:02,420 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:02,420 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/B, priority=13, startTime=1680490561993; duration=0sec 2023-04-03 02:56:02,421 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-03 02:56:02,421 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-03 02:56:02,422 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 02:56:02,422 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 02:56:02,422 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. because compaction request was cancelled 2023-04-03 02:56:02,422 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:56:02,423 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61074 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:56:02,423 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/C is initiating minor compaction (all files) 2023-04-03 02:56:02,423 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/C in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,423 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c8b01a2261ec4c2b879240b5b713d3bd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d06bfbe8bb054e62b0edf093fba3aeb1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/476aaf5a7a684c2ca4aeeefecd6f0158, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/f0057dfde4164aa5966be242656f0472, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7dca4342f8924b849c3355691c4f3c02] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=59.6 K 2023-04-03 02:56:02,424 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting c8b01a2261ec4c2b879240b5b713d3bd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680490559709 2023-04-03 02:56:02,424 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d06bfbe8bb054e62b0edf093fba3aeb1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680490560233 2023-04-03 02:56:02,425 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 476aaf5a7a684c2ca4aeeefecd6f0158, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680490560913 2023-04-03 02:56:02,425 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting f0057dfde4164aa5966be242656f0472, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680490561558 2023-04-03 02:56:02,425 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/3424337edbbc41da94354dee79f20a7c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3424337edbbc41da94354dee79f20a7c 2023-04-03 02:56:02,426 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7dca4342f8924b849c3355691c4f3c02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680490562188 2023-04-03 02:56:02,431 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/A of 9abe279a88a55aefb769568e98ff0ef5 into 3424337edbbc41da94354dee79f20a7c(size=12.4 K), total size for store is 41 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:02,431 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:02,431 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/A, priority=13, startTime=1680490561993; duration=0sec 2023-04-03 02:56:02,431 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 02:56:02,431 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 5 compacting, 0 eligible, 16 blocking 2023-04-03 02:56:02,431 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 02:56:02,431 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 02:56:02,432 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. because compaction request was cancelled 2023-04-03 02:56:02,432 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:02,433 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:02,433 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/B is initiating minor compaction (all files) 2023-04-03 02:56:02,433 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/B in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,433 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/80894bd081374f12828ba6e3440a26b8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8ab35e192f4148c3a0fe153c1e503640, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1a7dea6b390542f8a8e9e38d547a4945] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=36.2 K 2023-04-03 02:56:02,434 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 80894bd081374f12828ba6e3440a26b8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680490560913 2023-04-03 02:56:02,434 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 8ab35e192f4148c3a0fe153c1e503640, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680490561558 2023-04-03 02:56:02,435 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 1a7dea6b390542f8a8e9e38d547a4945, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680490562188 2023-04-03 02:56:02,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490622426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490622426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490622426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490622427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,438 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#C#compaction#467 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:02,440 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#B#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:02,452 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/73b5d9374d8944d89f46f8f253db56c5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/73b5d9374d8944d89f46f8f253db56c5 2023-04-03 02:56:02,456 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/C of 9abe279a88a55aefb769568e98ff0ef5 into 73b5d9374d8944d89f46f8f253db56c5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:02,456 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:02,456 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/C, priority=11, startTime=1680490562268; duration=0sec 2023-04-03 02:56:02,456 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:02,465 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/52c7cae976614ebda70dbf6ddf78745b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/52c7cae976614ebda70dbf6ddf78745b 2023-04-03 02:56:02,469 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/B of 9abe279a88a55aefb769568e98ff0ef5 into 52c7cae976614ebda70dbf6ddf78745b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:02,470 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:02,470 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/B, priority=13, startTime=1680490562268; duration=0sec 2023-04-03 02:56:02,470 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:02,514 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:02,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:02,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:02,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:02,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:02,524 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:02,524 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,524 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:02,524 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:02,524 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,525 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,526 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,526 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:02,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:02,526 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,526 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:02,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:02,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:02,540 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:02,541 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,541 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:02,542 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:02,542 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:02,542 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,544 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:02,544 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:02,544 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,544 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:02,546 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,547 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,548 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43277183[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:02,557 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:02,557 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,558 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:02,561 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:02,561 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:02,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,561 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:02,564 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:02,564 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:02,564 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:02,564 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:02,564 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:02,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:02,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,569 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,571 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,573 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,573 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,573 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,573 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:02,573 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:02,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:02,573 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,573 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:02,574 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:02,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:02,574 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:02,574 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:02,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:02,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:02,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490622637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490622638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490622638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490622638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,674 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:02,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:02,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:02,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:02,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:02,677 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:02,677 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,677 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:02,677 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:02,678 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,678 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:02,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:02,679 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:02,680 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,680 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:02,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:02,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:02,681 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,681 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:02,681 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:02,681 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:02,684 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,684 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,684 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,684 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:02,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dad0619[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:02,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:02,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,686 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:02,688 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:02,690 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:02,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:02,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:02,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:02,691 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:02,692 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:02,692 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,692 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:02,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:02,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:02,693 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,694 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/b3d6441f5afd48b7b759b5ec0fd8fd19 2023-04-03 02:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:02,705 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,705 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,705 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,705 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,705 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,705 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:02,705 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,705 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:02,705 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:02,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:02,705 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:02,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:02,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:02,705 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:02,706 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:02,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:02,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:02,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/5dee288189434c8689dec302b2d08bf3 2023-04-03 02:56:02,728 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/b95658d403d6447b918eb9c9e3ca6d73 2023-04-03 02:56:02,731 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/b3d6441f5afd48b7b759b5ec0fd8fd19 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/b3d6441f5afd48b7b759b5ec0fd8fd19 2023-04-03 02:56:02,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/b3d6441f5afd48b7b759b5ec0fd8fd19, entries=200, sequenceid=328, filesize=14.3 K 2023-04-03 02:56:02,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/5dee288189434c8689dec302b2d08bf3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5dee288189434c8689dec302b2d08bf3 2023-04-03 02:56:02,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5dee288189434c8689dec302b2d08bf3, entries=150, sequenceid=328, filesize=11.9 K 2023-04-03 02:56:02,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/b95658d403d6447b918eb9c9e3ca6d73 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b95658d403d6447b918eb9c9e3ca6d73 2023-04-03 02:56:02,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b95658d403d6447b918eb9c9e3ca6d73, entries=150, sequenceid=328, filesize=11.9 K 2023-04-03 02:56:02,746 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~240.80 KB/246576, currentSize=93.93 KB/96180 for 9abe279a88a55aefb769568e98ff0ef5 in 473ms, sequenceid=328, compaction requested=true 2023-04-03 02:56:02,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:02,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:02,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:02,746 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:02,746 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-03 02:56:02,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:02,747 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 02:56:02,747 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 02:56:02,748 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. because compaction request was cancelled 2023-04-03 02:56:02,748 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-03 02:56:02,748 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56610 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:02,748 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/A is initiating minor compaction (all files) 2023-04-03 02:56:02,748 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/A in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,748 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3424337edbbc41da94354dee79f20a7c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ce2cc877b43a4e03a788d88dd73f463e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8f87704b24ba4e34a8ee8abf920a939b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/b3d6441f5afd48b7b759b5ec0fd8fd19] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=55.3 K 2023-04-03 02:56:02,748 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 02:56:02,748 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 02:56:02,748 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 3424337edbbc41da94354dee79f20a7c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680490560913 2023-04-03 02:56:02,748 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. because compaction request was cancelled 2023-04-03 02:56:02,748 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting ce2cc877b43a4e03a788d88dd73f463e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680490561558 2023-04-03 02:56:02,749 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 8f87704b24ba4e34a8ee8abf920a939b, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680490562188 2023-04-03 02:56:02,749 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting b3d6441f5afd48b7b759b5ec0fd8fd19, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680490562234 2023-04-03 02:56:02,754 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#A#compaction#471 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:02,806 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:02,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:02,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:02,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:02,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:02,810 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:02,810 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,810 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:02,810 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:02,810 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,810 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:02,811 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,812 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,812 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:02,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:02,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:02,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:02,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:02,813 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:02,813 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:02,813 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 02:56:02,813 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:02,813 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:02,813 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:02,813 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:02,813 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:02,813 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:02,830 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/d7efd022e13f4bb8987be085e5f5ec70 2023-04-03 02:56:02,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:02,946 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:02,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490622963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490622964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490622965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:02,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:02,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490622965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490623071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490623072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490623075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490623076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490623147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,155 DEBUG [Thread-3082] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8204 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:56:03,176 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/cfb5f6f3b3df47e3880f72d94206b37c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/cfb5f6f3b3df47e3880f72d94206b37c 2023-04-03 02:56:03,182 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/A of 9abe279a88a55aefb769568e98ff0ef5 into cfb5f6f3b3df47e3880f72d94206b37c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:03,183 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:03,183 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/A, priority=12, startTime=1680490562746; duration=0sec 2023-04-03 02:56:03,183 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:03,242 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/a66046a929dc4b12b01d65c340e326c5 2023-04-03 02:56:03,261 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/0a2f1622062d437f95bedea824220929 2023-04-03 02:56:03,266 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/d7efd022e13f4bb8987be085e5f5ec70 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d7efd022e13f4bb8987be085e5f5ec70 2023-04-03 02:56:03,269 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d7efd022e13f4bb8987be085e5f5ec70, entries=150, sequenceid=349, filesize=11.9 K 2023-04-03 02:56:03,270 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/a66046a929dc4b12b01d65c340e326c5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a66046a929dc4b12b01d65c340e326c5 2023-04-03 02:56:03,273 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a66046a929dc4b12b01d65c340e326c5, entries=150, sequenceid=349, filesize=11.9 K 2023-04-03 02:56:03,274 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/0a2f1622062d437f95bedea824220929 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/0a2f1622062d437f95bedea824220929 2023-04-03 02:56:03,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490623275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490623277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,279 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/0a2f1622062d437f95bedea824220929, entries=150, sequenceid=349, filesize=11.9 K 2023-04-03 02:56:03,280 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 9abe279a88a55aefb769568e98ff0ef5 in 467ms, sequenceid=349, compaction requested=true 2023-04-03 02:56:03,280 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:03,280 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:03,280 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:03,280 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:03,280 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,280 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:03,280 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:03,282 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ae3f7df[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:03,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:03,284 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:03,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:03,287 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 02:56:03,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:03,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:03,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:03,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:03,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:03,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:03,287 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,287 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:03,290 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:03,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:03,295 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:03,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:03,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:03,295 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:03,295 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:03,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,297 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:03,298 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,300 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,302 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,302 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,302 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,302 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,302 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:03,302 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:03,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:03,302 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:03,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:03,303 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,303 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:03,303 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:03,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:03,303 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:03,304 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:03,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490623342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490623348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,404 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:03,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:03,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:03,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:03,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:03,409 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:03,409 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:03,409 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:03,409 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:03,409 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,410 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,411 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:03,411 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,411 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,411 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:03,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:03,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:03,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:03,428 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:03,428 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:03,428 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:03,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:03,436 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:03,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:03,437 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:03,437 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:03,437 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:03,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:03,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:03,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:03,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:03,440 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,440 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:03,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@112dc48d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:03,442 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:03,442 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,443 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,443 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:03,443 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:03,443 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:03,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:03,445 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:03,445 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:03,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,445 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,446 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:03,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:03,448 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:03,448 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:03,448 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:03,448 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:03,448 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:03,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,450 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490623450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,450 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:03,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:03,451 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,452 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,453 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,456 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,456 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,456 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,456 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,456 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:03,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:03,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:03,456 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,457 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,457 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,457 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:03,457 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:03,456 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:03,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:03,457 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:03,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:03,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:03,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490623459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,557 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:03,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:03,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:03,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:03,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:03,560 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:03,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:03,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:03,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:03,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,562 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:03,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:03,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,563 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,563 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:03,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:03,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:03,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:03,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:03,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:03,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:03,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:03,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:03,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:03,564 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:03,565 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:03,565 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:03,565 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:03,565 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:03,565 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:03,565 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,565 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:03,565 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:03,567 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,567 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:03,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:03,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69fe1442[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:03,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:03,570 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,571 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,571 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:03,571 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:03,571 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:03,572 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,572 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:03,575 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:03,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:03,577 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:03,577 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:03,577 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:03,577 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:03,577 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:03,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,580 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:03,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:03,581 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,583 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,585 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490623580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490623581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,589 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,589 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,589 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,589 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,589 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,589 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,589 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,589 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:03,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:03,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:03,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:03,589 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:03,589 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:03,589 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:03,589 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:03,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:03,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:03,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490623651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:03,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35310 deadline: 1680490623664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,690 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:03,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:03,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:03,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:03,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:03,693 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:03,693 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:03,693 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:03,693 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:03,693 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,693 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,694 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,694 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:03,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:03,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:03,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:03,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:03,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:03,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:03,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:03,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:03,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:03,696 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:03,696 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:03,696 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:03,696 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:03,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:03,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:03,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:03,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:03,697 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/184c135a71394f35a8f8a0969c13cf76 2023-04-03 02:56:03,699 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,699 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,699 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:03,700 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:03,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c665aaa[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:03,702 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:03,702 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,703 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:03,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:03,703 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:03,704 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,704 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:03,708 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/3e6ca87803f74dabab2fb08ab754de07 2023-04-03 02:56:03,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:03,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:03,710 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:03,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:03,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:03,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:03,710 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:03,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,717 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:03,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,718 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,721 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/2b59cb4da315453b98a4635aa26a7d8e 2023-04-03 02:56:03,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:03,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,723 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:03,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:03,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:03,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:03,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:03,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:03,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:03,723 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:03,724 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:03,726 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/184c135a71394f35a8f8a0969c13cf76 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/184c135a71394f35a8f8a0969c13cf76 2023-04-03 02:56:03,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/184c135a71394f35a8f8a0969c13cf76, entries=100, sequenceid=367, filesize=9.5 K 2023-04-03 02:56:03,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/3e6ca87803f74dabab2fb08ab754de07 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/3e6ca87803f74dabab2fb08ab754de07 2023-04-03 02:56:03,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/3e6ca87803f74dabab2fb08ab754de07, entries=100, sequenceid=367, filesize=9.5 K 2023-04-03 02:56:03,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/2b59cb4da315453b98a4635aa26a7d8e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2b59cb4da315453b98a4635aa26a7d8e 2023-04-03 02:56:03,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2b59cb4da315453b98a4635aa26a7d8e, entries=100, sequenceid=367, filesize=9.5 K 2023-04-03 02:56:03,736 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 9abe279a88a55aefb769568e98ff0ef5 in 450ms, sequenceid=367, compaction requested=true 2023-04-03 02:56:03,736 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:03,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:03,736 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:03,736 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:03,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:03,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:03,737 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:03,737 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/A is initiating minor compaction (all files) 2023-04-03 02:56:03,738 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/A in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:03,738 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/cfb5f6f3b3df47e3880f72d94206b37c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d7efd022e13f4bb8987be085e5f5ec70, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/184c135a71394f35a8f8a0969c13cf76] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=34.0 K 2023-04-03 02:56:03,738 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46948 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:03,738 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/B is initiating minor compaction (all files) 2023-04-03 02:56:03,738 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/B in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:03,738 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/52c7cae976614ebda70dbf6ddf78745b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5dee288189434c8689dec302b2d08bf3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a66046a929dc4b12b01d65c340e326c5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/3e6ca87803f74dabab2fb08ab754de07] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=45.8 K 2023-04-03 02:56:03,738 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting cfb5f6f3b3df47e3880f72d94206b37c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680490562259 2023-04-03 02:56:03,739 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 52c7cae976614ebda70dbf6ddf78745b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680490562188 2023-04-03 02:56:03,739 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting d7efd022e13f4bb8987be085e5f5ec70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680490562310 2023-04-03 02:56:03,739 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 5dee288189434c8689dec302b2d08bf3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680490562259 2023-04-03 02:56:03,739 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 184c135a71394f35a8f8a0969c13cf76, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680490562964 2023-04-03 02:56:03,739 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a66046a929dc4b12b01d65c340e326c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680490562310 2023-04-03 02:56:03,740 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 3e6ca87803f74dabab2fb08ab754de07, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680490562964 2023-04-03 02:56:03,745 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#A#compaction#478 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:03,749 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#B#compaction#479 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:03,754 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/913dbc2d6edb41198271faff880d730d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/913dbc2d6edb41198271faff880d730d 2023-04-03 02:56:03,757 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/A of 9abe279a88a55aefb769568e98ff0ef5 into 913dbc2d6edb41198271faff880d730d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:03,758 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:03,758 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/A, priority=13, startTime=1680490563736; duration=0sec 2023-04-03 02:56:03,758 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:03,758 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:03,759 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:03,759 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/C is initiating minor compaction (all files) 2023-04-03 02:56:03,759 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/C in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:03,759 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/73b5d9374d8944d89f46f8f253db56c5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b95658d403d6447b918eb9c9e3ca6d73, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/0a2f1622062d437f95bedea824220929, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2b59cb4da315453b98a4635aa26a7d8e] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=45.8 K 2023-04-03 02:56:03,760 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 73b5d9374d8944d89f46f8f253db56c5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680490562188 2023-04-03 02:56:03,760 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting b95658d403d6447b918eb9c9e3ca6d73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680490562259 2023-04-03 02:56:03,760 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 0a2f1622062d437f95bedea824220929, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680490562310 2023-04-03 02:56:03,761 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 2b59cb4da315453b98a4635aa26a7d8e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680490562964 2023-04-03 02:56:03,763 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/6545110422a245bb9e318d31f1a04c96 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/6545110422a245bb9e318d31f1a04c96 2023-04-03 02:56:03,767 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#C#compaction#480 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:03,767 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/B of 9abe279a88a55aefb769568e98ff0ef5 into 6545110422a245bb9e318d31f1a04c96(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:03,768 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:03,768 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/B, priority=12, startTime=1680490563736; duration=0sec 2023-04-03 02:56:03,768 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:03,781 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/eb25caccc949435ca06fddd335c6dcb7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/eb25caccc949435ca06fddd335c6dcb7 2023-04-03 02:56:03,784 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/C of 9abe279a88a55aefb769568e98ff0ef5 into eb25caccc949435ca06fddd335c6dcb7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:03,784 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:03,784 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/C, priority=12, startTime=1680490563737; duration=0sec 2023-04-03 02:56:03,784 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:03,824 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:03,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:03,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:03,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:03,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:03,827 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:03,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:03,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:03,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:03,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,828 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,829 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:03,829 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:03,829 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,829 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:03,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:03,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:03,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:03,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:03,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:03,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:03,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:03,831 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:03,831 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:03,831 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:03,831 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:03,832 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 02:56:03,832 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:03,832 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:03,832 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:03,832 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:03,832 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:03,832 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:03,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:03,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:04,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:04,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490624008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:04,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490624013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:04,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490624089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:04,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490624089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:04,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35336 deadline: 1680490624116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:04,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490624116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,242 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/c262ec71c87d4503adc57f4a6e8bd703 2023-04-03 02:56:04,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:04,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490624319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:04,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490624324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:04,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490624622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:04,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490624632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,652 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/2d085895b4d9458ba7c84068adb3368e 2023-04-03 02:56:04,660 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/9bfaf0f0e1034e2499c0eebfd1ace613 2023-04-03 02:56:04,665 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/c262ec71c87d4503adc57f4a6e8bd703 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c262ec71c87d4503adc57f4a6e8bd703 2023-04-03 02:56:04,667 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c262ec71c87d4503adc57f4a6e8bd703, entries=150, sequenceid=385, filesize=11.9 K 2023-04-03 02:56:04,669 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/2d085895b4d9458ba7c84068adb3368e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2d085895b4d9458ba7c84068adb3368e 2023-04-03 02:56:04,671 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2d085895b4d9458ba7c84068adb3368e, entries=150, sequenceid=385, filesize=11.9 K 2023-04-03 02:56:04,672 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/9bfaf0f0e1034e2499c0eebfd1ace613 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/9bfaf0f0e1034e2499c0eebfd1ace613 2023-04-03 02:56:04,675 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/9bfaf0f0e1034e2499c0eebfd1ace613, entries=150, sequenceid=385, filesize=11.9 K 2023-04-03 02:56:04,676 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 9abe279a88a55aefb769568e98ff0ef5 in 844ms, sequenceid=385, compaction requested=false 2023-04-03 02:56:04,676 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:04,676 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:04,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:04,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:04,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:04,676 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:04,677 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,677 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:04,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:04,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:04,678 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:04,678 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:04,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:04,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:04,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:04,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:04,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:04,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9de15f6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:04,679 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:04,679 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:04,681 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:04,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:04,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:04,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:04,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:04,681 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:04,681 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,681 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:04,682 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:04,682 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,682 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:04,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:04,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:04,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:04,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:04,685 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:04,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:04,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:04,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:04,685 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:04,686 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:04,686 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:04,686 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:04,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:04,687 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:04,688 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:04,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:04,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:04,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:04,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:04,690 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:04,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:04,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:04,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:04,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:04,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:04,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:04,691 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:04,691 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:04,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:04,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:04,791 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:04,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:04,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:04,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:04,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:04,794 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:04,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:04,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:04,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:04,795 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:04,795 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:04,797 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:04,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:04,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:04,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:04,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:04,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:04,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:04,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:04,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:04,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:04,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:04,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:04,798 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:04,799 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:04,799 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:04,799 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:04,799 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 02:56:04,799 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:04,799 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:04,799 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:04,799 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:04,799 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:04,799 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:05,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:05,098 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:05,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490625127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490625129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490625131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490625136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,208 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/a3ea4ba7413444f79e7b7909ab7f802e 2023-04-03 02:56:05,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490625231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490625233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490625238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490625438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490625439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490625449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,633 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/b8efbde2a943499b87f72e2dc73f6c70 2023-04-03 02:56:05,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490625742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490625745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:05,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:05,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490625753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,042 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/c2df2e4768304a9aa443fe209ddf73d8 2023-04-03 02:56:06,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/a3ea4ba7413444f79e7b7909ab7f802e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/a3ea4ba7413444f79e7b7909ab7f802e 2023-04-03 02:56:06,048 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/a3ea4ba7413444f79e7b7909ab7f802e, entries=150, sequenceid=402, filesize=11.9 K 2023-04-03 02:56:06,049 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/b8efbde2a943499b87f72e2dc73f6c70 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b8efbde2a943499b87f72e2dc73f6c70 2023-04-03 02:56:06,053 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b8efbde2a943499b87f72e2dc73f6c70, entries=150, sequenceid=402, filesize=11.9 K 2023-04-03 02:56:06,054 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/c2df2e4768304a9aa443fe209ddf73d8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c2df2e4768304a9aa443fe209ddf73d8 2023-04-03 02:56:06,057 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c2df2e4768304a9aa443fe209ddf73d8, entries=150, sequenceid=402, filesize=11.9 K 2023-04-03 02:56:06,057 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 9abe279a88a55aefb769568e98ff0ef5 in 1258ms, sequenceid=402, compaction requested=true 2023-04-03 02:56:06,057 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:06,058 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:06,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:06,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:06,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:06,059 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,059 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:06,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7628b1ec[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:06,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:06,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,062 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,062 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:06,062 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:06,062 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:06,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:06,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:06,064 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,064 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:06,064 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:06,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:06,066 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:06,066 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:06,066 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:06,066 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:06,066 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:06,068 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,068 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,068 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,069 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,070 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,072 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:06,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:06,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,073 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,073 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:06,073 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,073 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:06,073 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,073 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,073 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:06,152 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 02:56:06,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:06,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:06,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:06,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:06,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:06,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:06,173 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:06,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:06,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:06,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:06,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:06,176 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:06,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:06,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:06,177 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,177 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,178 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,178 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,178 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:06,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:06,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:06,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:06,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:06,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:06,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:06,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:06,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:06,180 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,180 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:06,180 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:06,180 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:06,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:06,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:06,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:06,181 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,181 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:06,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6df76f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:06,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:06,183 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,185 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:06,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:06,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:06,185 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:06,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,186 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:06,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:06,189 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:06,189 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:06,189 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:06,189 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:06,189 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:06,190 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,190 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,190 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,191 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,192 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:06,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:06,200 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,200 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:06,201 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:06,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:06,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,201 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:06,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490626230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:06,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490626247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:06,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35298 deadline: 1680490626253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:06,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490626260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,301 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:06,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:06,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:06,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:06,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:06,304 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:06,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:06,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:06,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,305 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:06,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:06,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:06,307 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,307 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:06,307 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:06,307 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:06,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,310 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:06,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31005322[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:06,310 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:06,310 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,311 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:06,312 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:06,312 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:06,312 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:06,314 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:06,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:06,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:06,316 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:06,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:06,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:06,317 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:06,317 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:06,318 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,318 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,318 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,318 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,320 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,321 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:06,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:06,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,322 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,322 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:06,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,322 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:06,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:06,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490626335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,422 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:06,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:06,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:06,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:06,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:06,425 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:06,426 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,426 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:06,426 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:06,426 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,426 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,427 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,427 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,428 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,428 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:06,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:06,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:06,429 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,429 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:06,429 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:06,429 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:06,432 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,432 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,432 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,432 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:06,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48888151[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:06,434 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:06,434 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,435 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,435 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:06,435 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:06,435 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:06,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,435 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:06,436 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:06,436 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,437 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:06,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:06,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:06,440 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:06,440 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:06,440 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:06,440 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:06,440 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:06,441 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,441 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,441 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,442 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,443 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,446 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,446 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,446 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,446 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,446 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,446 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:06,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:06,446 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,446 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,446 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,447 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:06,447 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:06,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:06,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,447 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:06,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490626540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,547 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:06,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:06,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:06,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:06,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:06,550 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:06,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:06,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:06,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,550 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,551 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,552 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,552 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,552 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:06,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:06,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:06,553 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,553 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:06,553 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:06,553 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:06,555 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,555 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,555 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,555 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9cee7d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:06,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:06,557 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:06,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:06,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/3fd17389fcb046ccaa4694db406dcabd 2023-04-03 02:56:06,564 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:06,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:06,566 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:06,566 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:06,566 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:06,566 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:06,566 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:06,568 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,568 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,568 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,569 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,578 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,581 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,581 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,581 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,581 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,582 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,581 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,582 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,582 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,582 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,582 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:06,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:06,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,582 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:06,582 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:06,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:06,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/765db5a0117743daa07619eabaa2f788 2023-04-03 02:56:06,682 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:06,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:06,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:06,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:06,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:06,685 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:06,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:06,685 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:06,686 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,686 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,690 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,690 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:06,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:06,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:06,692 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,692 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:06,692 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:06,692 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:06,692 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,693 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:06,693 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:06,694 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,694 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,694 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,694 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:06,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bcab8ee[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:06,696 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:06,696 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,697 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:06,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:06,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:06,698 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,698 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:06,700 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,700 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:06,700 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,701 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:06,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:06,704 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:06,704 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:06,704 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:06,704 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:06,704 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:06,705 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,707 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,708 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,711 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,711 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,711 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:06,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:06,712 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,712 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:06,712 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:06,713 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,713 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,713 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:06,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,812 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:06,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:06,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:06,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:06,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:06,815 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:06,815 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,815 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:06,815 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:06,815 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,815 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,816 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,817 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,817 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:06,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:06,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:06,818 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:06,818 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,818 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:06,818 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:06,818 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:06,818 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,818 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:06,818 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:06,818 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,818 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:06,819 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:06,819 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,819 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:06,819 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:06,821 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,821 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,821 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:06,822 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:06,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48607df6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:06,823 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:06,823 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,824 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,825 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:06,825 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:06,825 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:06,826 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,826 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:06,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:06,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,827 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,827 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:06,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:06,829 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:06,829 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:06,829 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:06,829 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:06,829 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:06,831 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,831 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,831 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,831 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,832 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,836 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,837 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:06,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:06,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:06,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,837 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:06,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,837 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:06,837 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:06,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490626845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,937 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:06,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:06,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:06,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:06,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:06,941 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:06,941 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,941 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:06,941 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:06,941 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,941 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,942 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,943 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,943 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:06,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:06,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:06,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:06,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:06,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:06,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:06,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:06,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:06,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:06,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:06,944 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:06,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:06,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:06,944 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:06,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:06,946 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,946 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,946 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,946 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:06,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446eefa2[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:06,948 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:06,948 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,949 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,949 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:06,949 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:06,949 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:06,949 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,949 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:06,954 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,954 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:06,954 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,954 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:06,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:06,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:06,956 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:06,956 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:06,956 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:06,956 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:06,956 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:06,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,958 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:06,958 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:06,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:06,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:06,960 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:06,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:06,962 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,962 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,962 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:06,962 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:06,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:06,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:06,962 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:06,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:06,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:06,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:06,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:06,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:06,962 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:06,962 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:06,962 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:06,962 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:06,963 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,963 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:06,963 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:07,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/a68614752a234ebeb797addbcbc015f3 2023-04-03 02:56:07,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/3fd17389fcb046ccaa4694db406dcabd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3fd17389fcb046ccaa4694db406dcabd 2023-04-03 02:56:07,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3fd17389fcb046ccaa4694db406dcabd, entries=200, sequenceid=418, filesize=14.3 K 2023-04-03 02:56:07,029 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/765db5a0117743daa07619eabaa2f788 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/765db5a0117743daa07619eabaa2f788 2023-04-03 02:56:07,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/765db5a0117743daa07619eabaa2f788, entries=150, sequenceid=418, filesize=11.9 K 2023-04-03 02:56:07,033 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/a68614752a234ebeb797addbcbc015f3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a68614752a234ebeb797addbcbc015f3 2023-04-03 02:56:07,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a68614752a234ebeb797addbcbc015f3, entries=150, sequenceid=418, filesize=11.9 K 2023-04-03 02:56:07,037 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 9abe279a88a55aefb769568e98ff0ef5 in 885ms, sequenceid=418, compaction requested=true 2023-04-03 02:56:07,037 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:07,037 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:07,037 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:07,037 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:07,037 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:07,037 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:07,041 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:07,041 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/B is initiating minor compaction (all files) 2023-04-03 02:56:07,041 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/B in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:07,041 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/6545110422a245bb9e318d31f1a04c96, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2d085895b4d9458ba7c84068adb3368e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b8efbde2a943499b87f72e2dc73f6c70, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/765db5a0117743daa07619eabaa2f788] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=48.4 K 2023-04-03 02:56:07,041 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51968 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:07,041 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 6545110422a245bb9e318d31f1a04c96, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680490562310 2023-04-03 02:56:07,041 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/A is initiating minor compaction (all files) 2023-04-03 02:56:07,042 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/A in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:07,042 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/913dbc2d6edb41198271faff880d730d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c262ec71c87d4503adc57f4a6e8bd703, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/a3ea4ba7413444f79e7b7909ab7f802e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3fd17389fcb046ccaa4694db406dcabd] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=50.8 K 2023-04-03 02:56:07,042 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 2d085895b4d9458ba7c84068adb3368e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680490563320 2023-04-03 02:56:07,043 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 913dbc2d6edb41198271faff880d730d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680490562310 2023-04-03 02:56:07,043 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b8efbde2a943499b87f72e2dc73f6c70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680490564002 2023-04-03 02:56:07,043 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c262ec71c87d4503adc57f4a6e8bd703, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680490563320 2023-04-03 02:56:07,043 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 765db5a0117743daa07619eabaa2f788, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1680490565126 2023-04-03 02:56:07,043 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a3ea4ba7413444f79e7b7909ab7f802e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680490564002 2023-04-03 02:56:07,044 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 3fd17389fcb046ccaa4694db406dcabd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1680490565126 2023-04-03 02:56:07,053 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#B#compaction#490 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:07,053 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#A#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:07,062 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:07,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:07,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:07,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:07,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:07,066 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:07,066 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:07,066 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:07,066 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:07,066 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:07,066 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:07,067 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:07,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:07,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:07,068 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/99aabcbe6f764727b40fd409e6d350a9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/99aabcbe6f764727b40fd409e6d350a9 2023-04-03 02:56:07,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:07,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:07,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:07,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:07,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:07,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:07,069 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:07,069 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,069 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:07,069 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:07,069 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:07,070 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:07,070 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:07,070 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:07,070 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:07,070 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 02:56:07,070 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:07,070 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:07,070 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:07,070 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:07,070 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:07,070 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:07,072 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/B of 9abe279a88a55aefb769568e98ff0ef5 into 99aabcbe6f764727b40fd409e6d350a9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:07,072 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:07,072 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/B, priority=12, startTime=1680490567037; duration=0sec 2023-04-03 02:56:07,072 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:07,072 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:07,075 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:07,075 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/C is initiating minor compaction (all files) 2023-04-03 02:56:07,075 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/C in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:07,075 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/eb25caccc949435ca06fddd335c6dcb7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/9bfaf0f0e1034e2499c0eebfd1ace613, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c2df2e4768304a9aa443fe209ddf73d8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a68614752a234ebeb797addbcbc015f3] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=48.3 K 2023-04-03 02:56:07,075 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting eb25caccc949435ca06fddd335c6dcb7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680490562310 2023-04-03 02:56:07,076 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 9bfaf0f0e1034e2499c0eebfd1ace613, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680490563320 2023-04-03 02:56:07,076 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting c2df2e4768304a9aa443fe209ddf73d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680490564002 2023-04-03 02:56:07,076 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a68614752a234ebeb797addbcbc015f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1680490565126 2023-04-03 02:56:07,116 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#C#compaction#493 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:07,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:07,257 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:07,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490627285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490627293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490627298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490627352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490627395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490627406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490627406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,468 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/6af6ebfc81bb46e5b76dcabe5fdd3fb1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/6af6ebfc81bb46e5b76dcabe5fdd3fb1 2023-04-03 02:56:07,475 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/A of 9abe279a88a55aefb769568e98ff0ef5 into 6af6ebfc81bb46e5b76dcabe5fdd3fb1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:07,475 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:07,475 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/A, priority=12, startTime=1680490567037; duration=0sec 2023-04-03 02:56:07,475 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:07,515 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/fb705c06a2154eef9bf5f224cd4d47a2 2023-04-03 02:56:07,534 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/f7683525288a4a419a4dda17360ff237 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/f7683525288a4a419a4dda17360ff237 2023-04-03 02:56:07,546 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/C of 9abe279a88a55aefb769568e98ff0ef5 into f7683525288a4a419a4dda17360ff237(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:07,547 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:07,547 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/C, priority=12, startTime=1680490567037; duration=0sec 2023-04-03 02:56:07,547 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:07,558 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/cb27d39b936e4ce683a1c780d0f86ad8 2023-04-03 02:56:07,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490627602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490627615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490627615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490627910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490627921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:07,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490627922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,977 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/c68f37a3982a4af3b456cb37503fbecc 2023-04-03 02:56:07,981 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/fb705c06a2154eef9bf5f224cd4d47a2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/fb705c06a2154eef9bf5f224cd4d47a2 2023-04-03 02:56:07,984 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/fb705c06a2154eef9bf5f224cd4d47a2, entries=150, sequenceid=434, filesize=11.9 K 2023-04-03 02:56:07,984 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/cb27d39b936e4ce683a1c780d0f86ad8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/cb27d39b936e4ce683a1c780d0f86ad8 2023-04-03 02:56:07,988 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/cb27d39b936e4ce683a1c780d0f86ad8, entries=150, sequenceid=434, filesize=11.9 K 2023-04-03 02:56:07,989 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/c68f37a3982a4af3b456cb37503fbecc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c68f37a3982a4af3b456cb37503fbecc 2023-04-03 02:56:07,992 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c68f37a3982a4af3b456cb37503fbecc, entries=150, sequenceid=434, filesize=11.9 K 2023-04-03 02:56:07,995 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 9abe279a88a55aefb769568e98ff0ef5 in 925ms, sequenceid=434, compaction requested=false 2023-04-03 02:56:07,995 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:07,995 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:07,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:07,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:07,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:07,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:07,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:08,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,000 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:08,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,000 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:08,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:08,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@581d0e54[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:08,002 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:08,002 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,003 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,003 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:08,003 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:08,003 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:08,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:08,007 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:08,007 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:08,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:08,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:08,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:08,010 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:08,010 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:08,010 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:08,010 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:08,010 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:08,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:08,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,014 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:08,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:08,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:08,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:08,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:08,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:08,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:08,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:08,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,020 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:08,020 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:08,020 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:08,020 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:08,121 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:08,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:08,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:08,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:08,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:08,124 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:08,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:08,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:08,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:08,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:08,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,126 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:08,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:08,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:08,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:08,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:08,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:08,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:08,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:08,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:08,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:08,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:08,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:08,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:08,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:08,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:08,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:08,128 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 02:56:08,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:08,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:08,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:08,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:08,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:08,129 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:08,144 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/1fa24aaba0494bbaa1972bdb831ac73c 2023-04-03 02:56:08,186 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/ec4a5c1cd980429fbbbc2ea0a92a3cc1 2023-04-03 02:56:08,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:08,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:08,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490628452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490628452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490628452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490628453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490628564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490628565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490628565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490628565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,607 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/66a0ff0e6c204bfe88bdc3f013aca00e 2023-04-03 02:56:08,610 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/1fa24aaba0494bbaa1972bdb831ac73c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/1fa24aaba0494bbaa1972bdb831ac73c 2023-04-03 02:56:08,614 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/1fa24aaba0494bbaa1972bdb831ac73c, entries=150, sequenceid=453, filesize=11.9 K 2023-04-03 02:56:08,615 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/ec4a5c1cd980429fbbbc2ea0a92a3cc1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4a5c1cd980429fbbbc2ea0a92a3cc1 2023-04-03 02:56:08,618 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4a5c1cd980429fbbbc2ea0a92a3cc1, entries=150, sequenceid=453, filesize=11.9 K 2023-04-03 02:56:08,618 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/66a0ff0e6c204bfe88bdc3f013aca00e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/66a0ff0e6c204bfe88bdc3f013aca00e 2023-04-03 02:56:08,621 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/66a0ff0e6c204bfe88bdc3f013aca00e, entries=150, sequenceid=453, filesize=11.9 K 2023-04-03 02:56:08,622 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 9abe279a88a55aefb769568e98ff0ef5 in 494ms, sequenceid=453, compaction requested=true 2023-04-03 02:56:08,622 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:08,622 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:08,622 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:08,622 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:08,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:08,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:08,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,628 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:08,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:08,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:08,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:08,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:08,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:08,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:08,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76bed071[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:08,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:08,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,633 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,633 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:08,633 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:08,633 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:08,634 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,634 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:08,636 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:08,636 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:08,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:08,636 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:08,637 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:08,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:08,639 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:08,639 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:08,640 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:08,640 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:08,640 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:08,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:08,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,641 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:08,644 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:08,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,646 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:08,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,649 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:08,649 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:08,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:08,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:08,650 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,650 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:08,650 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:08,650 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,650 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,650 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:08,650 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,650 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:08,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:08,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:08,650 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:08,650 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:08,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:08,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:08,750 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:08,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:08,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:08,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:08,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:08,756 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:08,757 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:08,757 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:08,757 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:08,757 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,757 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:08,759 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:08,759 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:08,760 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,760 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:08,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:08,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:08,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:08,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:08,761 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:08,761 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:08,761 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:08,764 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:08,764 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:08,764 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:08,764 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:08,764 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 02:56:08,765 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:08,765 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:08,765 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:08,765 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:08,765 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:08,765 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:08,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:08,774 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:08,786 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/e4c86246cccd4ebc8eb6c5d5c4455084 2023-04-03 02:56:08,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490628789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490628790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490628791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490628800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490628901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490628902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490628902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:08,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:08,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490628908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:09,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490629113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:09,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490629113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:09,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490629114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:09,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490629116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,218 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/85fc66c9930e4f74a75784ddb268eb6d 2023-04-03 02:56:09,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:09,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490629422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:09,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490629423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:09,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490629424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:09,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490629426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,638 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/3abbd784683941c2a0c0ce20287c1c17 2023-04-03 02:56:09,643 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/e4c86246cccd4ebc8eb6c5d5c4455084 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e4c86246cccd4ebc8eb6c5d5c4455084 2023-04-03 02:56:09,650 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e4c86246cccd4ebc8eb6c5d5c4455084, entries=150, sequenceid=470, filesize=11.9 K 2023-04-03 02:56:09,650 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/85fc66c9930e4f74a75784ddb268eb6d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/85fc66c9930e4f74a75784ddb268eb6d 2023-04-03 02:56:09,654 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/85fc66c9930e4f74a75784ddb268eb6d, entries=150, sequenceid=470, filesize=11.9 K 2023-04-03 02:56:09,654 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/3abbd784683941c2a0c0ce20287c1c17 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/3abbd784683941c2a0c0ce20287c1c17 2023-04-03 02:56:09,661 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/3abbd784683941c2a0c0ce20287c1c17, entries=150, sequenceid=470, filesize=11.9 K 2023-04-03 02:56:09,662 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 9abe279a88a55aefb769568e98ff0ef5 in 898ms, sequenceid=470, compaction requested=true 2023-04-03 02:56:09,662 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:09,662 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:09,662 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:09,662 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:09,662 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,664 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:09,664 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:09,665 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,665 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:09,666 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,666 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:09,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:09,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:09,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:09,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dfa9c9d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:09,668 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:09,668 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,669 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,669 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:09,669 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:09,669 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:09,670 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:09,670 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:09,670 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:09,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:09,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:09,671 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:09,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:09,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:09,671 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:09,671 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:09,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:09,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:09,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:09,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,681 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:09,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:09,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:09,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,683 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:09,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:09,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,685 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:09,685 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:09,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:09,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:09,686 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,686 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:09,686 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:09,686 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:09,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:09,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:09,686 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:09,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:09,686 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,686 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,686 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,686 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:09,686 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:09,787 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:09,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:09,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:09,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:09,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:09,790 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:09,790 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:09,790 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:09,790 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:09,791 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,791 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:09,792 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:09,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:09,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:09,792 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:09,792 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,792 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:09,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:09,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:09,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:09,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:09,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:09,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:09,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:09,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:09,796 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:09,796 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:09,796 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:09,796 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 02:56:09,796 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:09,797 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:09,797 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:09,797 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:09,797 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:09,797 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:09,830 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/ee06b3bb58e848c9a1f3ccc23c45e03a 2023-04-03 02:56:09,857 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/24cf6be1c2d140f68d7087fbeccff3a9 2023-04-03 02:56:09,878 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/daf3e9e099244956a5e5e44a764af554 2023-04-03 02:56:09,883 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/ee06b3bb58e848c9a1f3ccc23c45e03a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ee06b3bb58e848c9a1f3ccc23c45e03a 2023-04-03 02:56:09,886 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ee06b3bb58e848c9a1f3ccc23c45e03a, entries=150, sequenceid=485, filesize=11.9 K 2023-04-03 02:56:09,887 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/24cf6be1c2d140f68d7087fbeccff3a9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/24cf6be1c2d140f68d7087fbeccff3a9 2023-04-03 02:56:09,890 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/24cf6be1c2d140f68d7087fbeccff3a9, entries=150, sequenceid=485, filesize=11.9 K 2023-04-03 02:56:09,896 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/daf3e9e099244956a5e5e44a764af554 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/daf3e9e099244956a5e5e44a764af554 2023-04-03 02:56:09,899 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/daf3e9e099244956a5e5e44a764af554, entries=150, sequenceid=485, filesize=11.9 K 2023-04-03 02:56:09,900 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 9abe279a88a55aefb769568e98ff0ef5 in 104ms, sequenceid=485, compaction requested=true 2023-04-03 02:56:09,900 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:09,900 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:09,902 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,902 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:09,902 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,902 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:09,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:09,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eda677f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:09,905 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:09,905 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,908 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,908 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:09,908 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:09,908 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:09,908 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,908 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:09,912 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:09,913 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:09,913 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:09,913 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:09,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:09,916 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:09,916 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:09,916 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:09,916 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:09,916 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:09,918 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,918 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:09,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,920 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:09,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:09,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:09,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:09,922 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,926 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:09,926 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:09,926 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:09,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:09,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:09,926 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,927 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:09,927 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:09,927 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:09,927 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,927 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:09,927 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:09,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:09,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:09,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:09,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:09,927 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:09,927 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:09,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:09,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:09,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:56:09,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:09,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:09,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:09,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:09,943 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:09,943 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:09,951 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/c04eda19b7744c11a17ad3f7a331805b 2023-04-03 02:56:10,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490570003, max timestamp=1680490570014Num uniques 859; 2023-04-03 02:56:10,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:56:10,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490570003, max timestamp=1680490570014Num uniques 859; 2023-04-03 02:56:10,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:56:10,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490570003, max timestamp=1680490570014Num uniques 859; 2023-04-03 02:56:10,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:56:10,027 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:10,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:10,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:10,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:10,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:10,042 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:10,042 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:10,042 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:10,042 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:10,042 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,042 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,043 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,043 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:10,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:10,044 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,044 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:10,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:10,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:10,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:10,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:10,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:10,045 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:10,046 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,047 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:10,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:10,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e85b75e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:10,049 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:10,049 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,050 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,050 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:10,053 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,053 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:10,053 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:10,053 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:10,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:10,056 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:10,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:10,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:10,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:10,056 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:10,058 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,058 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,058 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:10,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:10,058 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:10,060 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,062 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:10,062 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:10,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:10,062 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,062 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,062 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,062 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:10,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:10,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:10,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:10,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:10,062 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:10,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,063 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:10,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,063 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:10,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490630068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490630069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490630070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490630071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,163 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:10,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:10,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:10,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:10,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:10,166 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:10,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:10,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:10,166 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:10,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,167 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,168 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:10,168 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:10,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:10,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:10,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:10,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:10,169 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:10,170 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,170 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:10,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:10,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:10,172 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:10,172 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:10,172 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:10,172 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:10,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:10,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:10,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:10,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:10,174 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:10,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:10,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:10,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e38c5a7[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:10,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:10,176 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,178 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:10,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:10,178 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:10,178 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,178 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:10,179 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:10,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:10,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490630175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490630180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490630180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:10,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:10,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490630181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,186 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:10,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:10,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:10,186 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:10,186 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:10,194 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,194 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,194 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:10,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,194 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:10,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:10,195 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:10,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:10,199 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:10,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:10,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:10,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:10,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,200 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,200 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:10,200 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:10,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:10,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:10,300 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:10,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:10,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:10,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:10,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:10,303 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:10,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:10,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:10,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:10,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,304 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,305 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,305 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:10,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:10,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:10,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,306 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:10,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:10,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:10,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:10,307 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:10,307 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:10,308 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:10,308 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:10,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:10,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:10,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:10,308 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:10,309 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,309 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:10,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:10,310 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,310 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:10,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:10,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@574fa889[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:10,312 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:10,313 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,314 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:10,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:10,314 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:10,314 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,314 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:10,316 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:10,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:10,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:10,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:10,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:10,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:10,319 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:10,319 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:10,319 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:10,319 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:10,319 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:10,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:10,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,325 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,326 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,328 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:10,329 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:10,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:10,329 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,329 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,329 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,329 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,329 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:10,329 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,329 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:10,329 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:10,330 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:10,330 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:10,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:10,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:10,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:10,368 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/76dea5f3642647b6a9bfb3f3e166c40a 2023-04-03 02:56:10,383 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/1065842b754741fdbeaf0216474bc2d3 2023-04-03 02:56:10,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/c04eda19b7744c11a17ad3f7a331805b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c04eda19b7744c11a17ad3f7a331805b 2023-04-03 02:56:10,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490630387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490630392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490630392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490630393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c04eda19b7744c11a17ad3f7a331805b, entries=200, sequenceid=495, filesize=14.3 K 2023-04-03 02:56:10,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/76dea5f3642647b6a9bfb3f3e166c40a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/76dea5f3642647b6a9bfb3f3e166c40a 2023-04-03 02:56:10,402 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/76dea5f3642647b6a9bfb3f3e166c40a, entries=150, sequenceid=495, filesize=11.9 K 2023-04-03 02:56:10,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/1065842b754741fdbeaf0216474bc2d3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/1065842b754741fdbeaf0216474bc2d3 2023-04-03 02:56:10,415 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/1065842b754741fdbeaf0216474bc2d3, entries=150, sequenceid=495, filesize=11.9 K 2023-04-03 02:56:10,416 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9abe279a88a55aefb769568e98ff0ef5 in 474ms, sequenceid=495, compaction requested=true 2023-04-03 02:56:10,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:10,416 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:10,416 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 02:56:10,417 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 02:56:10,416 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:10,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:10,429 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:10,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:10,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:10,435 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74036 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 02:56:10,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:10,435 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/B is initiating minor compaction (all files) 2023-04-03 02:56:10,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:10,435 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/B in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:10,435 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:10,436 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/99aabcbe6f764727b40fd409e6d350a9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/cb27d39b936e4ce683a1c780d0f86ad8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4a5c1cd980429fbbbc2ea0a92a3cc1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/85fc66c9930e4f74a75784ddb268eb6d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/24cf6be1c2d140f68d7087fbeccff3a9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/76dea5f3642647b6a9bfb3f3e166c40a] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=72.3 K 2023-04-03 02:56:10,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:10,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:10,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:10,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,436 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,438 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,438 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:10,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:10,439 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,439 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:10,439 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76476 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 02:56:10,439 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 99aabcbe6f764727b40fd409e6d350a9, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1680490565126 2023-04-03 02:56:10,439 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/A is initiating minor compaction (all files) 2023-04-03 02:56:10,440 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/A in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:10,440 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/6af6ebfc81bb46e5b76dcabe5fdd3fb1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/fb705c06a2154eef9bf5f224cd4d47a2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/1fa24aaba0494bbaa1972bdb831ac73c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e4c86246cccd4ebc8eb6c5d5c4455084, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ee06b3bb58e848c9a1f3ccc23c45e03a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c04eda19b7744c11a17ad3f7a331805b] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=74.7 K 2023-04-03 02:56:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:10,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:10,440 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting cb27d39b936e4ce683a1c780d0f86ad8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680490566215 2023-04-03 02:56:10,440 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6af6ebfc81bb46e5b76dcabe5fdd3fb1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1680490565126 2023-04-03 02:56:10,440 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ec4a5c1cd980429fbbbc2ea0a92a3cc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1680490567274 2023-04-03 02:56:10,441 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting fb705c06a2154eef9bf5f224cd4d47a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680490566215 2023-04-03 02:56:10,441 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 85fc66c9930e4f74a75784ddb268eb6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680490568425 2023-04-03 02:56:10,441 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 1fa24aaba0494bbaa1972bdb831ac73c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1680490567274 2023-04-03 02:56:10,441 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 24cf6be1c2d140f68d7087fbeccff3a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680490568787 2023-04-03 02:56:10,441 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e4c86246cccd4ebc8eb6c5d5c4455084, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680490568425 2023-04-03 02:56:10,442 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 76dea5f3642647b6a9bfb3f3e166c40a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680490569935 2023-04-03 02:56:10,442 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting ee06b3bb58e848c9a1f3ccc23c45e03a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680490568787 2023-04-03 02:56:10,442 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c04eda19b7744c11a17ad3f7a331805b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680490569935 2023-04-03 02:56:10,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:10,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:10,444 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:10,444 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:10,444 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:10,461 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:10,461 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:10,461 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 02:56:10,461 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:10,461 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:10,461 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:10,461 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:10,461 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:10,461 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:10,469 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#A#compaction#508 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:10,475 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#B#compaction#509 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:10,500 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/c8690c6a0148460d9de77b239ea63b2b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c8690c6a0148460d9de77b239ea63b2b 2023-04-03 02:56:10,505 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/256486dc0ce04ee5bce04eb03b94bcf3 2023-04-03 02:56:10,506 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/A of 9abe279a88a55aefb769568e98ff0ef5 into c8690c6a0148460d9de77b239ea63b2b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:10,506 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:10,506 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/A, priority=10, startTime=1680490570416; duration=0sec 2023-04-03 02:56:10,507 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:10,507 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 02:56:10,510 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74002 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 02:56:10,510 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/C is initiating minor compaction (all files) 2023-04-03 02:56:10,510 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/C in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:10,510 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/f7683525288a4a419a4dda17360ff237, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c68f37a3982a4af3b456cb37503fbecc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/66a0ff0e6c204bfe88bdc3f013aca00e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/3abbd784683941c2a0c0ce20287c1c17, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/daf3e9e099244956a5e5e44a764af554, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/1065842b754741fdbeaf0216474bc2d3] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=72.3 K 2023-04-03 02:56:10,511 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting f7683525288a4a419a4dda17360ff237, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1680490565126 2023-04-03 02:56:10,511 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c68f37a3982a4af3b456cb37503fbecc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680490566215 2023-04-03 02:56:10,512 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 66a0ff0e6c204bfe88bdc3f013aca00e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1680490567274 2023-04-03 02:56:10,512 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 3abbd784683941c2a0c0ce20287c1c17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680490568425 2023-04-03 02:56:10,512 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting daf3e9e099244956a5e5e44a764af554, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680490568787 2023-04-03 02:56:10,513 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 1065842b754741fdbeaf0216474bc2d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680490569935 2023-04-03 02:56:10,535 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#C#compaction#511 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:10,568 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/384b618dadde45b9b06f3d05786c45cc 2023-04-03 02:56:10,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:10,713 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:10,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490630726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490630727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490630728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490630728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490630839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490630840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490630840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:10,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490630848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:10,900 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/e413a96b8b7d4d569831db30e23d0f06 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/e413a96b8b7d4d569831db30e23d0f06 2023-04-03 02:56:10,906 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/B of 9abe279a88a55aefb769568e98ff0ef5 into e413a96b8b7d4d569831db30e23d0f06(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:10,906 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:10,906 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/B, priority=10, startTime=1680490570416; duration=0sec 2023-04-03 02:56:10,906 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:10,970 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/1396ba1f6d5840c2ac6c093fcaf88569 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/1396ba1f6d5840c2ac6c093fcaf88569 2023-04-03 02:56:10,974 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/C of 9abe279a88a55aefb769568e98ff0ef5 into 1396ba1f6d5840c2ac6c093fcaf88569(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:10,974 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:10,974 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/C, priority=10, startTime=1680490570417; duration=0sec 2023-04-03 02:56:10,974 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:10,989 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/ca723af6af11463cb704ebbb1427103a 2023-04-03 02:56:10,993 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/256486dc0ce04ee5bce04eb03b94bcf3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/256486dc0ce04ee5bce04eb03b94bcf3 2023-04-03 02:56:10,996 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/256486dc0ce04ee5bce04eb03b94bcf3, entries=150, sequenceid=532, filesize=11.9 K 2023-04-03 02:56:10,997 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/384b618dadde45b9b06f3d05786c45cc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/384b618dadde45b9b06f3d05786c45cc 2023-04-03 02:56:11,001 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/384b618dadde45b9b06f3d05786c45cc, entries=150, sequenceid=532, filesize=11.9 K 2023-04-03 02:56:11,002 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/ca723af6af11463cb704ebbb1427103a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/ca723af6af11463cb704ebbb1427103a 2023-04-03 02:56:11,007 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/ca723af6af11463cb704ebbb1427103a, entries=150, sequenceid=532, filesize=11.9 K 2023-04-03 02:56:11,008 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for 9abe279a88a55aefb769568e98ff0ef5 in 547ms, sequenceid=532, compaction requested=false 2023-04-03 02:56:11,008 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:11,008 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:11,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:11,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:11,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:11,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:11,010 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,010 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,010 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@af3dc80[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:11,012 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:11,012 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:11,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:11,013 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:11,014 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,014 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:11,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:11,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:11,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:11,016 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:11,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:11,019 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:11,019 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:11,019 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:11,019 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:11,019 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:11,038 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,038 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:11,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,038 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,038 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:11,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:11,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:11,042 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:11,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:11,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:11,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:11,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:11,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:11,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:11,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:11,043 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:11,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:11,043 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:11,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:11,055 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:56:11,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:11,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:11,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:11,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:11,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:11,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:11,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/3815e1bdfc4f45faa0f3d165f1e8d08f 2023-04-03 02:56:11,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/a21cf059664f4d5fa217a7eb33d0c2b2 2023-04-03 02:56:11,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490631088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680490571072, max timestamp=1680490571097Num uniques 859; 2023-04-03 02:56:11,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:56:11,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680490571072, max timestamp=1680490571097Num uniques 859; 2023-04-03 02:56:11,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=219, cellSize=10024, totalHeapSize=23016, min timestamp=1680490571072, max timestamp=1680490571097Num uniques 859; 2023-04-03 02:56:11,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 02:56:11,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=10024, suffix heap size=117064,new segment heap size=23016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=219 2023-04-03 02:56:11,143 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:11,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:11,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:11,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490631149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:11,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:11,156 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:11,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:11,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:11,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:11,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,157 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,158 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:11,158 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,158 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,158 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:11,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:11,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490631150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490631153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:11,168 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:11,168 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:11,168 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:11,172 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:11,172 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:11,172 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:11,172 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:11,175 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:11,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:11,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@195fa5c7[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:11,177 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:11,177 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,180 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:11,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:11,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:11,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:11,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:11,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:11,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:11,182 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:11,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:11,185 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:11,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:11,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:11,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:11,185 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:11,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:11,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:11,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:11,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,189 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:11,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:11,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:11,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:11,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:11,193 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:11,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:11,193 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,194 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:11,194 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:11,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490631205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490631254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490631263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490631267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,294 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:11,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:11,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:11,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:11,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:11,298 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:11,298 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:11,298 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:11,298 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:11,299 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,299 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,301 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:11,301 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:11,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:11,301 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,301 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:11,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:11,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:11,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:11,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:11,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:11,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:11,317 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:11,318 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:11,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:11,320 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,321 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,321 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,321 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b25eb6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:11,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:11,321 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,323 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,323 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,323 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:11,323 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:11,323 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:11,323 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:11,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:11,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:11,329 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:11,329 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:11,329 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:11,329 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:11,329 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:11,335 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,335 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:11,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,336 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,336 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,339 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,347 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:11,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:11,347 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:11,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:11,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:11,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:11,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:11,352 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:11,352 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:11,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490631411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,452 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:11,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:11,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:11,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:11,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:11,456 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:11,456 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:11,456 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:11,456 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:11,458 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,458 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,459 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,459 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,459 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:11,460 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:11,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:11,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:11,464 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:11,464 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:11,464 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:11,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490631462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,480 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:11,480 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:11,482 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:11,482 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:11,482 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:11,482 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:11,482 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:11,482 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:11,482 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,482 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:11,482 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:11,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490631477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490631477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/7a75a785862a49e58ebaf2a1f633e73f 2023-04-03 02:56:11,489 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,490 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,490 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,490 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:11,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/3815e1bdfc4f45faa0f3d165f1e8d08f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3815e1bdfc4f45faa0f3d165f1e8d08f 2023-04-03 02:56:11,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:11,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b93b9e9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:11,492 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:11,492 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,493 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:11,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:11,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:11,494 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,494 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:11,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:11,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:11,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:11,496 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,497 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3815e1bdfc4f45faa0f3d165f1e8d08f, entries=200, sequenceid=547, filesize=14.3 K 2023-04-03 02:56:11,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/a21cf059664f4d5fa217a7eb33d0c2b2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a21cf059664f4d5fa217a7eb33d0c2b2 2023-04-03 02:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:11,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:11,502 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:11,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:11,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:11,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:11,502 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:11,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:11,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:11,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:11,505 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,507 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a21cf059664f4d5fa217a7eb33d0c2b2, entries=150, sequenceid=547, filesize=11.9 K 2023-04-03 02:56:11,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,512 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:11,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:11,512 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:11,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:11,512 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,512 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:11,512 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:11,512 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:11,512 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:11,512 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/7a75a785862a49e58ebaf2a1f633e73f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7a75a785862a49e58ebaf2a1f633e73f 2023-04-03 02:56:11,512 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:11,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:11,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:11,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:11,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7a75a785862a49e58ebaf2a1f633e73f, entries=150, sequenceid=547, filesize=11.9 K 2023-04-03 02:56:11,521 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=125.68 KB/128700 for 9abe279a88a55aefb769568e98ff0ef5 in 466ms, sequenceid=547, compaction requested=true 2023-04-03 02:56:11,521 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:11,521 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:11,522 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:11,522 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:11,522 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:11,522 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:11,522 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:11,523 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/A is initiating minor compaction (all files) 2023-04-03 02:56:11,523 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:11,523 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/A in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:11,523 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c8690c6a0148460d9de77b239ea63b2b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/256486dc0ce04ee5bce04eb03b94bcf3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3815e1bdfc4f45faa0f3d165f1e8d08f] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=39.2 K 2023-04-03 02:56:11,523 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/B is initiating minor compaction (all files) 2023-04-03 02:56:11,523 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/B in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:11,523 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/e413a96b8b7d4d569831db30e23d0f06, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/384b618dadde45b9b06f3d05786c45cc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a21cf059664f4d5fa217a7eb33d0c2b2] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=36.8 K 2023-04-03 02:56:11,523 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c8690c6a0148460d9de77b239ea63b2b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680490569935 2023-04-03 02:56:11,523 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e413a96b8b7d4d569831db30e23d0f06, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680490569935 2023-04-03 02:56:11,524 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 256486dc0ce04ee5bce04eb03b94bcf3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680490570053 2023-04-03 02:56:11,524 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 384b618dadde45b9b06f3d05786c45cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680490570053 2023-04-03 02:56:11,524 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 3815e1bdfc4f45faa0f3d165f1e8d08f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680490570714 2023-04-03 02:56:11,524 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a21cf059664f4d5fa217a7eb33d0c2b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680490570714 2023-04-03 02:56:11,532 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#A#compaction#517 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:11,537 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#B#compaction#518 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:11,580 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/ecdc985daee24368aa1781026ec88d0f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ecdc985daee24368aa1781026ec88d0f 2023-04-03 02:56:11,583 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/76686f7beeb646348fe1f75fb8118cc8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/76686f7beeb646348fe1f75fb8118cc8 2023-04-03 02:56:11,587 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/B of 9abe279a88a55aefb769568e98ff0ef5 into ecdc985daee24368aa1781026ec88d0f(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:11,588 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:11,588 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/B, priority=13, startTime=1680490571522; duration=0sec 2023-04-03 02:56:11,588 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:11,588 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:11,589 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/A of 9abe279a88a55aefb769568e98ff0ef5 into 76686f7beeb646348fe1f75fb8118cc8(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:11,589 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:11,589 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/A, priority=13, startTime=1680490571521; duration=0sec 2023-04-03 02:56:11,589 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:11,590 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:11,592 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/C is initiating minor compaction (all files) 2023-04-03 02:56:11,592 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/C in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:11,592 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/1396ba1f6d5840c2ac6c093fcaf88569, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/ca723af6af11463cb704ebbb1427103a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7a75a785862a49e58ebaf2a1f633e73f] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=36.8 K 2023-04-03 02:56:11,592 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 1396ba1f6d5840c2ac6c093fcaf88569, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1680490569935 2023-04-03 02:56:11,593 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ca723af6af11463cb704ebbb1427103a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680490570053 2023-04-03 02:56:11,593 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7a75a785862a49e58ebaf2a1f633e73f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680490570714 2023-04-03 02:56:11,599 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#C#compaction#519 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:11,612 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:11,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:11,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:11,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:11,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:11,622 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:11,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:11,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:11,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:11,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:11,632 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:11,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:11,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:11,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:11,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:11,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:11,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:11,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:11,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:11,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:11,638 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/e3d8b0cc3e0948aaad251af1f5616e04 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e3d8b0cc3e0948aaad251af1f5616e04 2023-04-03 02:56:11,646 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/C of 9abe279a88a55aefb769568e98ff0ef5 into e3d8b0cc3e0948aaad251af1f5616e04(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:11,646 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:11,646 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/C, priority=13, startTime=1680490571522; duration=0sec 2023-04-03 02:56:11,646 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:11,647 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:11,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:11,648 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:11,648 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=125.68 KB heapSize=351.63 KB 2023-04-03 02:56:11,648 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:11,648 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:11,648 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:11,648 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:11,648 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:11,648 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:11,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:11,722 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:11,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490631778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490631788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490631788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490631789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490631889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:11,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:11,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490631898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,072 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/8d6be7f1a61342a6a168f4cccd8c1a21 2023-04-03 02:56:12,094 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/8e1bacc1a2a4487fa48d43ae451e2830 2023-04-03 02:56:12,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:12,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490632094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:12,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490632105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:12,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490632301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:12,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490632301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:12,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490632402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:12,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490632413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,529 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.70 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/e898f2a5f5a341f48af83574d82896c5 2023-04-03 02:56:12,533 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/8d6be7f1a61342a6a168f4cccd8c1a21 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8d6be7f1a61342a6a168f4cccd8c1a21 2023-04-03 02:56:12,536 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8d6be7f1a61342a6a168f4cccd8c1a21, entries=150, sequenceid=584, filesize=11.9 K 2023-04-03 02:56:12,537 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/8e1bacc1a2a4487fa48d43ae451e2830 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8e1bacc1a2a4487fa48d43ae451e2830 2023-04-03 02:56:12,541 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8e1bacc1a2a4487fa48d43ae451e2830, entries=150, sequenceid=584, filesize=11.9 K 2023-04-03 02:56:12,542 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/e898f2a5f5a341f48af83574d82896c5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e898f2a5f5a341f48af83574d82896c5 2023-04-03 02:56:12,545 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e898f2a5f5a341f48af83574d82896c5, entries=150, sequenceid=584, filesize=11.9 K 2023-04-03 02:56:12,545 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.68 KB/128700, heapSize ~351.58 KB/360016, currentSize=60.38 KB/61830 for 9abe279a88a55aefb769568e98ff0ef5 in 897ms, sequenceid=584, compaction requested=false 2023-04-03 02:56:12,545 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:12,545 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:12,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:12,546 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:12,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,547 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:12,547 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,547 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:12,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:12,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f107bc7[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:12,549 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:12,549 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,550 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:12,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:12,550 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:12,551 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,551 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:12,553 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:12,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:12,556 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:12,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:12,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:12,556 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:12,556 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:12,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:12,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:12,558 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:12,559 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:12,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:12,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:12,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:12,561 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:12,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:12,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:12,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:12,561 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,562 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,562 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:12,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:12,562 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:12,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:12,662 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:12,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:12,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:12,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:12,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:12,667 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:12,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:12,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:12,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:12,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:12,672 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:12,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:12,672 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:12,673 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,673 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:12,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:12,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:12,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:12,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:12,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:12,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:12,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:12,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:12,674 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:12,674 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:12,674 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:12,674 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:56:12,674 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:12,675 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:12,675 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:12,675 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:12,675 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:12,675 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:12,689 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=596 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/f9fd590eb6354002944f310707203e18 2023-04-03 02:56:12,725 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=596 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/2aa1264af78d4cee871755325a0487f5 2023-04-03 02:56:12,749 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=596 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/21e14d4852c44657ad19dd0e740d3af7 2023-04-03 02:56:12,755 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/f9fd590eb6354002944f310707203e18 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/f9fd590eb6354002944f310707203e18 2023-04-03 02:56:12,762 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/f9fd590eb6354002944f310707203e18, entries=150, sequenceid=596, filesize=11.9 K 2023-04-03 02:56:12,762 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/2aa1264af78d4cee871755325a0487f5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2aa1264af78d4cee871755325a0487f5 2023-04-03 02:56:12,778 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2aa1264af78d4cee871755325a0487f5, entries=150, sequenceid=596, filesize=11.9 K 2023-04-03 02:56:12,779 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/21e14d4852c44657ad19dd0e740d3af7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/21e14d4852c44657ad19dd0e740d3af7 2023-04-03 02:56:12,788 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/21e14d4852c44657ad19dd0e740d3af7, entries=150, sequenceid=596, filesize=11.9 K 2023-04-03 02:56:12,789 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 9abe279a88a55aefb769568e98ff0ef5 in 115ms, sequenceid=596, compaction requested=true 2023-04-03 02:56:12,789 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:12,790 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:12,790 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:12,790 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:12,790 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,790 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:12,790 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:12,791 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,791 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:12,792 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,792 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:12,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:12,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:12,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:12,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b058c7f[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:12,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:12,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,795 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,795 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:12,796 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:12,799 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:12,799 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:12,799 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:12,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:12,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:12,804 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:12,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:12,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:12,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:12,804 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:12,807 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,807 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:12,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,807 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:12,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,811 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:12,812 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:12,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:12,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:12,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:12,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:12,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:12,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:12,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:12,817 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:12,817 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,817 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,817 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:12,817 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,816 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:12,817 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:12,817 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:12,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:12,917 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:12,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:12,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:12,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:12,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:12,920 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:12,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:12,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:12,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:12,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:12,923 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:12,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:12,924 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:12,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:12,924 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:12,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:12,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:12,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:12,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:12,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:12,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:12,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:12,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:12,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:12,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:12,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:12,925 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:12,925 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:12,925 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 02:56:12,925 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:12,925 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:12,926 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:12,926 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:12,926 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:12,926 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:12,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:12,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:12,951 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/4d5487d7107f4c9293a4cd5e3e2687e6 2023-04-03 02:56:12,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490572955, max timestamp=1680490572977Num uniques 859; 2023-04-03 02:56:12,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490572955, max timestamp=1680490572977Num uniques 859; 2023-04-03 02:56:12,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490572955, max timestamp=1680490572977Num uniques 859; 2023-04-03 02:56:12,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:12,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:12,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:13,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680490573032, max timestamp=1680490573047Num uniques 1068; 2023-04-03 02:56:13,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 02:56:13,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680490573032, max timestamp=1680490573047Num uniques 1068; 2023-04-03 02:56:13,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680490573032, max timestamp=1680490573047Num uniques 1068; 2023-04-03 02:56:13,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 02:56:13,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 02:56:13,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680490573097, max timestamp=1680490573127Num uniques 1077; 2023-04-03 02:56:13,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-03 02:56:13,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680490573097, max timestamp=1680490573127Num uniques 1077; 2023-04-03 02:56:13,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680490573097, max timestamp=1680490573127Num uniques 1077; 2023-04-03 02:56:13,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-03 02:56:13,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-03 02:56:13,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680490573187, max timestamp=1680490573199Num uniques 1086; 2023-04-03 02:56:13,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-03 02:56:13,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680490573187, max timestamp=1680490573199Num uniques 1086; 2023-04-03 02:56:13,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-03 02:56:13,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680490573187, max timestamp=1680490573199Num uniques 1086; 2023-04-03 02:56:13,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-03 02:56:13,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680490573230, max timestamp=1680490573237Num uniques 1045; 2023-04-03 02:56:13,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-03 02:56:13,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680490573230, max timestamp=1680490573237Num uniques 1045; 2023-04-03 02:56:13,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-03 02:56:13,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680490573230, max timestamp=1680490573237Num uniques 1045; 2023-04-03 02:56:13,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-03 02:56:13,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680490573260, max timestamp=1680490573275Num uniques 1054; 2023-04-03 02:56:13,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-03 02:56:13,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680490573260, max timestamp=1680490573275Num uniques 1054; 2023-04-03 02:56:13,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680490573260, max timestamp=1680490573275Num uniques 1054; 2023-04-03 02:56:13,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-03 02:56:13,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-03 02:56:13,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680490573299, max timestamp=1680490573318Num uniques 1013; 2023-04-03 02:56:13,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680490573299, max timestamp=1680490573318Num uniques 1013; 2023-04-03 02:56:13,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-03 02:56:13,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-03 02:56:13,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680490573299, max timestamp=1680490573318Num uniques 1013; 2023-04-03 02:56:13,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-03 02:56:13,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:13,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490633343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7864, totalHeapSize=18128, min timestamp=1680490573341, max timestamp=1680490573350Num uniques 1072; 2023-04-03 02:56:13,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49084, new segment data size=7864, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-03 02:56:13,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7870, totalHeapSize=18128, min timestamp=1680490573341, max timestamp=1680490573350Num uniques 1072; 2023-04-03 02:56:13,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49090, new segment data size=7870, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-03 02:56:13,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7869, totalHeapSize=18128, min timestamp=1680490573341, max timestamp=1680490573350Num uniques 1072; 2023-04-03 02:56:13,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49089, new segment data size=7869, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-03 02:56:13,366 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/ddbd030993484c6197aa449b9036f960 2023-04-03 02:56:13,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680490573368, max timestamp=1680490573391Num uniques 1031; 2023-04-03 02:56:13,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-03 02:56:13,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680490573368, max timestamp=1680490573391Num uniques 1031; 2023-04-03 02:56:13,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-03 02:56:13,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680490573376, max timestamp=1680490573391Num uniques 1031; 2023-04-03 02:56:13,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=6870, suffix heap size=135192,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=150 2023-04-03 02:56:13,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680490573433, max timestamp=1680490573445Num uniques 1140; 2023-04-03 02:56:13,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680490573433, max timestamp=1680490573445Num uniques 1140; 2023-04-03 02:56:13,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680490573433, max timestamp=1680490573445Num uniques 1009; 2023-04-03 02:56:13,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13280, suffix heap size=132904,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=290 2023-04-03 02:56:13,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-04-03 02:56:13,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-04-03 02:56:13,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680490573469, max timestamp=1680490573479Num uniques 1149; 2023-04-03 02:56:13,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-03 02:56:13,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680490573469, max timestamp=1680490573479Num uniques 1149; 2023-04-03 02:56:13,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-03 02:56:13,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680490573469, max timestamp=1680490573479Num uniques 1149; 2023-04-03 02:56:13,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-03 02:56:13,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:13,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490633510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:13,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490633511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680490573508, max timestamp=1680490573518Num uniques 1108; 2023-04-03 02:56:13,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680490573508, max timestamp=1680490573518Num uniques 1108; 2023-04-03 02:56:13,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-03 02:56:13,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-03 02:56:13,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680490573508, max timestamp=1680490573518Num uniques 1108; 2023-04-03 02:56:13,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-03 02:56:13,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680490573544, max timestamp=1680490573568Num uniques 1117; 2023-04-03 02:56:13,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680490573544, max timestamp=1680490573568Num uniques 1117; 2023-04-03 02:56:13,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-03 02:56:13,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680490573544, max timestamp=1680490573568Num uniques 1117; 2023-04-03 02:56:13,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-03 02:56:13,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-03 02:56:13,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680490573598, max timestamp=1680490573620Num uniques 1076; 2023-04-03 02:56:13,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-03 02:56:13,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680490573598, max timestamp=1680490573620Num uniques 1076; 2023-04-03 02:56:13,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680490573598, max timestamp=1680490573620Num uniques 1076; 2023-04-03 02:56:13,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-03 02:56:13,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-03 02:56:13,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680490573634, max timestamp=1680490573653Num uniques 1085; 2023-04-03 02:56:13,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=15340, suffix heap size=140808,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=335 2023-04-03 02:56:13,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680490573634, max timestamp=1680490573653Num uniques 1085; 2023-04-03 02:56:13,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=15340, suffix heap size=140808,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=335 2023-04-03 02:56:13,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680490573634, max timestamp=1680490573653Num uniques 1085; 2023-04-03 02:56:13,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=15340, suffix heap size=140808,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=335 2023-04-03 02:56:13,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=394, cellSize=18044, totalHeapSize=41216, min timestamp=1680490573665, max timestamp=1680490573686Num uniques 1194; 2023-04-03 02:56:13,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=18044, suffix heap size=152144,new segment heap size=41216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=394 2023-04-03 02:56:13,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=394, cellSize=18044, totalHeapSize=41216, min timestamp=1680490573665, max timestamp=1680490573686Num uniques 1194; 2023-04-03 02:56:13,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=394, cellSize=18044, totalHeapSize=41216, min timestamp=1680490573665, max timestamp=1680490573686Num uniques 1194; 2023-04-03 02:56:13,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=18044, suffix heap size=152144,new segment heap size=41216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=394 2023-04-03 02:56:13,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=18044, suffix heap size=152144,new segment heap size=41216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=394 2023-04-03 02:56:13,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:13,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490633723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:13,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490633724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:13,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490633725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:13,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490633726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:13,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490633727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,777 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/94f747a8199444dd915d7a9ee0c3f29a 2023-04-03 02:56:13,781 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/4d5487d7107f4c9293a4cd5e3e2687e6 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/4d5487d7107f4c9293a4cd5e3e2687e6 2023-04-03 02:56:13,784 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/4d5487d7107f4c9293a4cd5e3e2687e6, entries=150, sequenceid=603, filesize=11.9 K 2023-04-03 02:56:13,785 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/ddbd030993484c6197aa449b9036f960 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ddbd030993484c6197aa449b9036f960 2023-04-03 02:56:13,788 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ddbd030993484c6197aa449b9036f960, entries=150, sequenceid=603, filesize=11.9 K 2023-04-03 02:56:13,790 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/94f747a8199444dd915d7a9ee0c3f29a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/94f747a8199444dd915d7a9ee0c3f29a 2023-04-03 02:56:13,798 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/94f747a8199444dd915d7a9ee0c3f29a, entries=150, sequenceid=603, filesize=11.9 K 2023-04-03 02:56:13,800 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 9abe279a88a55aefb769568e98ff0ef5 in 875ms, sequenceid=603, compaction requested=true 2023-04-03 02:56:13,800 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:13,800 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:13,800 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:13,801 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,801 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:13,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,802 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:13,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:13,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b13247[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:13,803 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:13,803 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,804 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:13,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,804 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:13,804 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:13,806 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:13,806 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:13,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:13,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:13,808 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:13,808 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:13,808 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:13,809 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:13,809 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:13,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,811 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:13,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:13,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:13,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:13,811 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,813 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:13,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:13,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:13,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:13,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:13,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,815 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:13,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:13,815 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:13,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:13,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:13,815 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:13,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:13,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:13,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,815 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,816 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:13,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=167.72 KB heapSize=463.03 KB 2023-04-03 02:56:13,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680490573723, max timestamp=1680490573840Num uniques 1253; 2023-04-03 02:56:13,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=57385, new segment data size=9295, suffix heap size=158280,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1253, new segment cells count=203 2023-04-03 02:56:13,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680490573723, max timestamp=1680490573840Num uniques 1253; 2023-04-03 02:56:13,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=57385, new segment data size=9295, suffix heap size=158280,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1253, new segment cells count=203 2023-04-03 02:56:13,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680490573723, max timestamp=1680490573840Num uniques 1253; 2023-04-03 02:56:13,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=57385, new segment data size=9295, suffix heap size=158280,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1253, new segment cells count=203 2023-04-03 02:56:13,844 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:13,844 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:13,844 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:13,844 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:13,844 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:13,844 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:13,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=899 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/e0c1ac77b6094047bd67140c0a6ebbd5 2023-04-03 02:56:13,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490573859, max timestamp=1680490573880Num uniques 859; 2023-04-03 02:56:13,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490573859, max timestamp=1680490573880Num uniques 859; 2023-04-03 02:56:13,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490573859, max timestamp=1680490573880Num uniques 859; 2023-04-03 02:56:13,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:13,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:13,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:13,915 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:13,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:13,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490573894, max timestamp=1680490573914Num uniques 1068; 2023-04-03 02:56:13,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-03 02:56:13,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490573894, max timestamp=1680490573914Num uniques 1068; 2023-04-03 02:56:13,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680490573894, max timestamp=1680490573914Num uniques 1068; 2023-04-03 02:56:13,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-03 02:56:13,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-03 02:56:13,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:13,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:13,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:13,919 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:13,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:13,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:13,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:13,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:13,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,920 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:13,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:13,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:13,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:13,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:13,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:13,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:13,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:13,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:13,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:13,923 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:13,923 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:13,923 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:13,923 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:13,925 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:13,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:13,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:13,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3b9649[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:13,927 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:13,927 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,929 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,929 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,929 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:13,929 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:13,929 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:13,929 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:13,930 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:13,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:13,930 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:13,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:13,934 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:13,934 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:13,934 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:13,934 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:13,934 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:13,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:13,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:13,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:13,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:13,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:13,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:13,936 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:13,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,937 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:13,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,939 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:13,939 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:13,939 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:13,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:13,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:13,939 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:13,939 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:13,939 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:13,939 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:13,940 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:13,940 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,940 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:13,940 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:13,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:13,940 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:13,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680490573941, max timestamp=1680490573954Num uniques 1127; 2023-04-03 02:56:13,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680490573941, max timestamp=1680490573954Num uniques 1127; 2023-04-03 02:56:13,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-03 02:56:13,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-03 02:56:13,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680490573941, max timestamp=1680490573954Num uniques 1127; 2023-04-03 02:56:13,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-03 02:56:13,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:13,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490633989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680490573984, max timestamp=1680490573997Num uniques 1186; 2023-04-03 02:56:14,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=13096, suffix heap size=151312,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=286 2023-04-03 02:56:14,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680490573984, max timestamp=1680490573997Num uniques 1186; 2023-04-03 02:56:14,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680490573984, max timestamp=1680490573997Num uniques 1186; 2023-04-03 02:56:14,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=13096, suffix heap size=151312,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=286 2023-04-03 02:56:14,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=13096, suffix heap size=151312,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=286 2023-04-03 02:56:14,040 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:14,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:14,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:14,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490634039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680490574020, max timestamp=1680490574042Num uniques 1145; 2023-04-03 02:56:14,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-03 02:56:14,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680490574020, max timestamp=1680490574042Num uniques 1145; 2023-04-03 02:56:14,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680490574020, max timestamp=1680490574042Num uniques 1145; 2023-04-03 02:56:14,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:14,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-03 02:56:14,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:14,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-03 02:56:14,046 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:14,046 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,046 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:14,046 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:14,046 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,046 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,049 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,049 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,049 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,049 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:14,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:14,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:14,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:14,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:14,050 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:14,050 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,050 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:14,051 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,052 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:14,052 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:14,052 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:14,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:14,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:14,052 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:14,053 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,053 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,054 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:14,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a9080a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:14,055 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:14,055 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,056 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:14,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:14,056 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:14,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:14,058 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:14,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,058 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:14,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:14,060 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:14,060 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:14,060 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:14,060 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:14,061 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:14,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,063 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,065 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,066 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,068 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,068 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,068 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,068 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,068 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,068 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,068 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:14,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:14,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,068 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,069 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,069 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:14,069 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:14,068 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490634105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490634111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490634113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490634114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490634145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,169 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:14,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:14,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:14,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:14,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:14,172 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:14,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:14,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:14,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,173 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,173 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,174 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:14,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:14,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:14,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,179 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:14,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:14,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:14,180 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,180 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:14,180 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:14,180 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:14,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:14,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:14,180 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:14,186 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,186 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:14,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55580254[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:14,188 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:14,188 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:14,191 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,191 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:14,192 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:14,192 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:14,192 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,192 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:14,194 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,195 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:14,195 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,195 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:14,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:14,196 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:14,196 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:14,196 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:14,196 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:14,196 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:14,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,199 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,200 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,202 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,206 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,206 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,207 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,207 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:14,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:14,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:14,208 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:14,208 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:14,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490634216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490634220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490634222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=899 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/8f5742bd1af643aaa4f9e124a235ceb1 2023-04-03 02:56:14,308 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:14,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:14,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:14,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:14,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:14,316 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:14,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:14,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:14,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,316 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,318 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,318 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,318 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,318 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:14,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:14,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:14,328 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:14,328 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,328 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:14,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:14,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:14,333 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,333 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:14,333 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:14,333 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:14,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:14,333 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:14,334 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:14,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490634315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,335 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,335 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,336 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,336 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:14,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:14,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ba21b2[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:14,337 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:14,337 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,339 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,340 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:14,340 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:14,340 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:14,340 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,340 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:14,341 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,341 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:14,341 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,341 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:14,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:14,343 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:14,343 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:14,343 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:14,343 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:14,343 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:14,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,345 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,346 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,347 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490634349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,351 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,351 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,351 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,351 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:14,351 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:14,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:14,352 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:14,352 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:14,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,352 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,352 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,352 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,352 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,352 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490634424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490634425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490634431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,452 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:14,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:14,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:14,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:14,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:14,459 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:14,459 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,459 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:14,459 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:14,459 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,460 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,466 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,466 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,467 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,467 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:14,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:14,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:14,468 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:14,468 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,468 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:14,472 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:14,472 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:14,472 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,472 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:14,472 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:14,472 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,472 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:14,472 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:14,472 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,472 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:14,472 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:14,473 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,473 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,474 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,474 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:14,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@567164b2[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:14,475 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:14,475 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,477 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:14,477 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,477 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:14,480 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:14,480 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,480 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,480 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:14,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:14,482 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:14,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:14,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:14,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:14,483 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:14,485 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,485 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,485 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,486 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,487 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,489 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,489 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,489 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,489 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,489 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,490 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,490 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,490 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,490 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:14,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:14,490 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,490 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:14,490 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:14,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,590 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:14,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:14,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:14,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:14,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:14,593 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:14,593 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,593 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:14,593 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:14,594 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,594 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,595 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,595 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,595 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,595 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:14,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:14,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:14,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:14,597 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:14,597 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,597 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:14,597 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:14,597 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:14,597 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,597 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:14,597 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:14,598 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,598 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:14,598 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:14,598 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,598 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:14,598 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:14,600 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:14,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c12cf2e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:14,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:14,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,604 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:14,604 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,604 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:14,605 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,606 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:14,606 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,606 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:14,608 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:14,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:14,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:14,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:14,608 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:14,609 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,609 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,609 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,610 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,611 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,613 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,613 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,613 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:14,613 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:14,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:14,613 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,613 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,613 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,613 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,613 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,613 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:14,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,613 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:14,613 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490634638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490634651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=899 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/a310cc300ac24c62ae09020475d82180 2023-04-03 02:56:14,708 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/e0c1ac77b6094047bd67140c0a6ebbd5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e0c1ac77b6094047bd67140c0a6ebbd5 2023-04-03 02:56:14,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e0c1ac77b6094047bd67140c0a6ebbd5, entries=150, sequenceid=899, filesize=11.9 K 2023-04-03 02:56:14,712 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/8f5742bd1af643aaa4f9e124a235ceb1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8f5742bd1af643aaa4f9e124a235ceb1 2023-04-03 02:56:14,714 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:14,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:14,715 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8f5742bd1af643aaa4f9e124a235ceb1, entries=150, sequenceid=899, filesize=11.9 K 2023-04-03 02:56:14,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:14,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/a310cc300ac24c62ae09020475d82180 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a310cc300ac24c62ae09020475d82180 2023-04-03 02:56:14,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:14,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:14,717 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:14,717 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,717 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:14,717 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:14,717 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,717 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,719 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,720 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,720 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:14,720 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:14,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:14,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:14,724 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:14,724 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:14,724 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:14,724 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:14,724 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:14,724 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,724 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. started... 2023-04-03 02:56:14,725 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. as already flushing 2023-04-03 02:56:14,725 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,725 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:14,725 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:14,725 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,725 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:14,725 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:14,725 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a310cc300ac24c62ae09020475d82180, entries=150, sequenceid=899, filesize=11.9 K 2023-04-03 02:56:14,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~81.98 KB/83952, currentSize=154.31 KB/158010 for 9abe279a88a55aefb769568e98ff0ef5 in 886ms, sequenceid=899, compaction requested=true 2023-04-03 02:56:14,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:14,727 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:56:14,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:14,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:14,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:14,727 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:56:14,727 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,729 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,730 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,730 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,731 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62156 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:56:14,731 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/B is initiating minor compaction (all files) 2023-04-03 02:56:14,731 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/B in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,731 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ecdc985daee24368aa1781026ec88d0f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8e1bacc1a2a4487fa48d43ae451e2830, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2aa1264af78d4cee871755325a0487f5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ddbd030993484c6197aa449b9036f960, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8f5742bd1af643aaa4f9e124a235ceb1] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=60.7 K 2023-04-03 02:56:14,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:14,731 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62156 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:56:14,732 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/A is initiating minor compaction (all files) 2023-04-03 02:56:14,732 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/A in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,732 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/76686f7beeb646348fe1f75fb8118cc8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8d6be7f1a61342a6a168f4cccd8c1a21, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/f9fd590eb6354002944f310707203e18, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/4d5487d7107f4c9293a4cd5e3e2687e6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e0c1ac77b6094047bd67140c0a6ebbd5] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=60.7 K 2023-04-03 02:56:14,731 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:14,732 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49cd2c5a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:14,732 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ecdc985daee24368aa1781026ec88d0f, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680490570714 2023-04-03 02:56:14,732 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 76686f7beeb646348fe1f75fb8118cc8, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680490570714 2023-04-03 02:56:14,733 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 8e1bacc1a2a4487fa48d43ae451e2830, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1680490571146 2023-04-03 02:56:14,733 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 8d6be7f1a61342a6a168f4cccd8c1a21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1680490571146 2023-04-03 02:56:14,733 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:14,733 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,734 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:14,733 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 2aa1264af78d4cee871755325a0487f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=596, earliestPutTs=1680490571774 2023-04-03 02:56:14,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:14,733 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting f9fd590eb6354002944f310707203e18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=596, earliestPutTs=1680490571774 2023-04-03 02:56:14,734 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ddbd030993484c6197aa449b9036f960, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1680490572911 2023-04-03 02:56:14,734 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 4d5487d7107f4c9293a4cd5e3e2687e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1680490572911 2023-04-03 02:56:14,735 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e0c1ac77b6094047bd67140c0a6ebbd5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=899, earliestPutTs=1680490573723 2023-04-03 02:56:14,735 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 8f5742bd1af643aaa4f9e124a235ceb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=899, earliestPutTs=1680490573723 2023-04-03 02:56:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,736 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:14,736 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:14,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:14,741 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:14,741 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:14,741 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:14,741 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:14,741 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:14,744 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,744 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,745 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:14,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:14,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,748 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:14,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680490574092, max timestamp=1680490574749Num uniques 1154; 2023-04-03 02:56:14,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680490574092, max timestamp=1680490574749Num uniques 1154; 2023-04-03 02:56:14,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-03 02:56:14,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-03 02:56:14,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=71.56 KB heapSize=195.34 KB 2023-04-03 02:56:14,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:14,752 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:14,752 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:14,752 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:14,752 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:14,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:14,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:14,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 02:56:14,754 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:14,754 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:14,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:14,754 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,755 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,755 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:14,755 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,755 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,755 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:14,755 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:14,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:14,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:14,755 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:14,755 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#A#compaction#532 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:14,755 DEBUG [Thread-3084] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:14,755 DEBUG [Thread-3084] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:14,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:14,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:14,767 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#B#compaction#534 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:14,796 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/8b22819905d3408ea2330dd8999aaecc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8b22819905d3408ea2330dd8999aaecc 2023-04-03 02:56:14,800 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/A of 9abe279a88a55aefb769568e98ff0ef5 into 8b22819905d3408ea2330dd8999aaecc(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:14,800 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:14,800 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/A, priority=11, startTime=1680490574727; duration=0sec 2023-04-03 02:56:14,800 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:14,800 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:56:14,802 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62122 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:56:14,802 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 9abe279a88a55aefb769568e98ff0ef5/C is initiating minor compaction (all files) 2023-04-03 02:56:14,802 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9abe279a88a55aefb769568e98ff0ef5/C in TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:14,802 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e3d8b0cc3e0948aaad251af1f5616e04, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e898f2a5f5a341f48af83574d82896c5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/21e14d4852c44657ad19dd0e740d3af7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/94f747a8199444dd915d7a9ee0c3f29a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a310cc300ac24c62ae09020475d82180] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp, totalSize=60.7 K 2023-04-03 02:56:14,803 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e3d8b0cc3e0948aaad251af1f5616e04, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680490570714 2023-04-03 02:56:14,804 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e898f2a5f5a341f48af83574d82896c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1680490571146 2023-04-03 02:56:14,804 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 21e14d4852c44657ad19dd0e740d3af7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=596, earliestPutTs=1680490571774 2023-04-03 02:56:14,805 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 94f747a8199444dd915d7a9ee0c3f29a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1680490572911 2023-04-03 02:56:14,805 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a310cc300ac24c62ae09020475d82180, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=899, earliestPutTs=1680490573723 2023-04-03 02:56:14,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1006 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/703a89c4876e44a291ac087b5157db46 2023-04-03 02:56:14,818 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9abe279a88a55aefb769568e98ff0ef5#C#compaction#535 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:14,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490634805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490634812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490634819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,833 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/ecc54fe6691c41f7a3a35fd931a508f9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/ecc54fe6691c41f7a3a35fd931a508f9 2023-04-03 02:56:14,840 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/C of 9abe279a88a55aefb769568e98ff0ef5 into ecc54fe6691c41f7a3a35fd931a508f9(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:14,840 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:14,840 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/C, priority=11, startTime=1680490574727; duration=0sec 2023-04-03 02:56:14,840 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:14,852 DEBUG [Thread-3087] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:52506 2023-04-03 02:56:14,852 DEBUG [Thread-3087] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:14,855 DEBUG [Thread-3093] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65816b6e to 127.0.0.1:52506 2023-04-03 02:56:14,855 DEBUG [Thread-3093] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:14,855 DEBUG [Thread-3084] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:14,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:14,857 DEBUG [Thread-3089] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f5363d8 to 127.0.0.1:52506 2023-04-03 02:56:14,857 DEBUG [Thread-3089] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:14,860 DEBUG [Thread-3085] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:52506 2023-04-03 02:56:14,860 DEBUG [Thread-3085] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:14,860 DEBUG [Thread-3091] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x211582f2 to 127.0.0.1:52506 2023-04-03 02:56:14,861 DEBUG [Thread-3091] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:14,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490634920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490634923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:14,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:14,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490634933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:15,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:15,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490635121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:15,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:15,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490635124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:15,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:15,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490635135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:15,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:15,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35276 deadline: 1680490635146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:15,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:15,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35286 deadline: 1680490635160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:15,210 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/49e44cd405ad467cb3c16523dff1ebc9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/49e44cd405ad467cb3c16523dff1ebc9 2023-04-03 02:56:15,214 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9abe279a88a55aefb769568e98ff0ef5/B of 9abe279a88a55aefb769568e98ff0ef5 into 49e44cd405ad467cb3c16523dff1ebc9(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:15,214 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:15,214 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5., storeName=9abe279a88a55aefb769568e98ff0ef5/B, priority=11, startTime=1680490574727; duration=0sec 2023-04-03 02:56:15,214 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:15,236 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1006 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/eda4d61f774041baaa96c5e91a4e65f3 2023-04-03 02:56:15,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:15,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35310 deadline: 1680490635423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:15,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:15,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35336 deadline: 1680490635425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:15,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:15,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35298 deadline: 1680490635437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:15,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=1006 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/3cc3ae4ab3ba422b9455a052104b1e2b 2023-04-03 02:56:15,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/703a89c4876e44a291ac087b5157db46 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/703a89c4876e44a291ac087b5157db46 2023-04-03 02:56:15,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/703a89c4876e44a291ac087b5157db46, entries=200, sequenceid=1006, filesize=14.3 K 2023-04-03 02:56:15,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/eda4d61f774041baaa96c5e91a4e65f3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/eda4d61f774041baaa96c5e91a4e65f3 2023-04-03 02:56:15,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/eda4d61f774041baaa96c5e91a4e65f3, entries=150, sequenceid=1006, filesize=11.9 K 2023-04-03 02:56:15,658 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/3cc3ae4ab3ba422b9455a052104b1e2b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/3cc3ae4ab3ba422b9455a052104b1e2b 2023-04-03 02:56:15,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/3cc3ae4ab3ba422b9455a052104b1e2b, entries=150, sequenceid=1006, filesize=11.9 K 2023-04-03 02:56:15,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~195.30 KB/199984, currentSize=107.34 KB/109920 for 9abe279a88a55aefb769568e98ff0ef5 in 910ms, sequenceid=1006, compaction requested=false 2023-04-03 02:56:15,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:15,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:15,926 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 02:56:15,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:15,926 DEBUG [Thread-3076] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27f2d997 to 127.0.0.1:52506 2023-04-03 02:56:15,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:15,926 DEBUG [Thread-3076] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:15,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:15,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:15,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:15,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:15,931 DEBUG [Thread-3078] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34ad4aa8 to 127.0.0.1:52506 2023-04-03 02:56:15,931 DEBUG [Thread-3078] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:15,939 DEBUG [Thread-3080] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c29757d to 127.0.0.1:52506 2023-04-03 02:56:15,939 DEBUG [Thread-3080] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:16,150 DEBUG [Thread-3074] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x501c5a88 to 127.0.0.1:52506 2023-04-03 02:56:16,150 DEBUG [Thread-3074] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:16,163 DEBUG [Thread-3082] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06edd017 to 127.0.0.1:52506 2023-04-03 02:56:16,164 DEBUG [Thread-3082] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:16,164 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 02:56:16,164 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 229 2023-04-03 02:56:16,164 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 248 2023-04-03 02:56:16,164 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 165 2023-04-03 02:56:16,164 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 196 2023-04-03 02:56:16,164 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-04-03 02:56:16,164 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 02:56:16,164 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 02:56:16,164 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1790 2023-04-03 02:56:16,165 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5369 rows 2023-04-03 02:56:16,165 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1781 2023-04-03 02:56:16,165 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5337 rows 2023-04-03 02:56:16,165 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1781 2023-04-03 02:56:16,165 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5340 rows 2023-04-03 02:56:16,165 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1783 2023-04-03 02:56:16,165 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5348 rows 2023-04-03 02:56:16,165 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(402): scanned 1793 2023-04-03 02:56:16,165 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(403): verified 5376 rows 2023-04-03 02:56:16,165 INFO [Listener at localhost/46853] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 02:56:16,165 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e372785 to 127.0.0.1:52506 2023-04-03 02:56:16,165 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:16,167 INFO [Listener at localhost/46853] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 02:56:16,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 02:56:16,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:16,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 02:56:16,174 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490576174"}]},"ts":"1680490576174"} 2023-04-03 02:56:16,175 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 02:56:16,177 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 02:56:16,178 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9abe279a88a55aefb769568e98ff0ef5, UNASSIGN}] 2023-04-03 02:56:16,180 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9abe279a88a55aefb769568e98ff0ef5, UNASSIGN 2023-04-03 02:56:16,180 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=9abe279a88a55aefb769568e98ff0ef5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:16,180 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490576180"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490576180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490576180"}]},"ts":"1680490576180"} 2023-04-03 02:56:16,181 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:56:16,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 02:56:16,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:16,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9abe279a88a55aefb769568e98ff0ef5, disabling compactions & flushes 2023-04-03 02:56:16,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:16,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1029 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/f55c7c1bbdab4c538c1557730dbd948c 2023-04-03 02:56:16,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1029 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/1e4dce8fad5a49beb02bb28aa9ad6e8d 2023-04-03 02:56:16,359 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1029 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/32bfe153938945f4b5f48980db192279 2023-04-03 02:56:16,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/f55c7c1bbdab4c538c1557730dbd948c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/f55c7c1bbdab4c538c1557730dbd948c 2023-04-03 02:56:16,366 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/f55c7c1bbdab4c538c1557730dbd948c, entries=150, sequenceid=1029, filesize=11.9 K 2023-04-03 02:56:16,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/1e4dce8fad5a49beb02bb28aa9ad6e8d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1e4dce8fad5a49beb02bb28aa9ad6e8d 2023-04-03 02:56:16,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1e4dce8fad5a49beb02bb28aa9ad6e8d, entries=150, sequenceid=1029, filesize=11.9 K 2023-04-03 02:56:16,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/32bfe153938945f4b5f48980db192279 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/32bfe153938945f4b5f48980db192279 2023-04-03 02:56:16,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/32bfe153938945f4b5f48980db192279, entries=150, sequenceid=1029, filesize=11.9 K 2023-04-03 02:56:16,373 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=26.84 KB/27480 for 9abe279a88a55aefb769568e98ff0ef5 in 447ms, sequenceid=1029, compaction requested=true 2023-04-03 02:56:16,374 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:16,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:16,374 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. because compaction request was cancelled 2023-04-03 02:56:16,374 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:16,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:16,374 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. because compaction request was cancelled 2023-04-03 02:56:16,374 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:16,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. after waiting 0 ms 2023-04-03 02:56:16,374 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:16,374 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. because compaction request was cancelled 2023-04-03 02:56:16,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:16,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 9abe279a88a55aefb769568e98ff0ef5 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 02:56:16,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=A 2023-04-03 02:56:16,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:16,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=B 2023-04-03 02:56:16,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:16,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9abe279a88a55aefb769568e98ff0ef5, store=C 2023-04-03 02:56:16,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:16,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 02:56:16,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 02:56:16,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1036 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/7c68f8abde3f4743a0cc66276aa265f7 2023-04-03 02:56:16,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1036 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/d0f7aaacfe0e4a0fad3e713c7ad70730 2023-04-03 02:56:17,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1036 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/62a29c158c2048eaae49ff6553e37a74 2023-04-03 02:56:17,212 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/A/7c68f8abde3f4743a0cc66276aa265f7 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/7c68f8abde3f4743a0cc66276aa265f7 2023-04-03 02:56:17,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/7c68f8abde3f4743a0cc66276aa265f7, entries=150, sequenceid=1036, filesize=11.9 K 2023-04-03 02:56:17,216 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/B/d0f7aaacfe0e4a0fad3e713c7ad70730 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/d0f7aaacfe0e4a0fad3e713c7ad70730 2023-04-03 02:56:17,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/d0f7aaacfe0e4a0fad3e713c7ad70730, entries=150, sequenceid=1036, filesize=11.9 K 2023-04-03 02:56:17,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/.tmp/C/62a29c158c2048eaae49ff6553e37a74 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/62a29c158c2048eaae49ff6553e37a74 2023-04-03 02:56:17,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/62a29c158c2048eaae49ff6553e37a74, entries=150, sequenceid=1036, filesize=11.9 K 2023-04-03 02:56:17,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 9abe279a88a55aefb769568e98ff0ef5 in 850ms, sequenceid=1036, compaction requested=true 2023-04-03 02:56:17,226 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/32f84934fd0a4bd9a720c1ca4d2803cb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/2434ab635f8045e292d93db2d16eca92, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e076428bec3d4feb98c861427706bee2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/84d5a088fbfd4d6e916beb9984407c0f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/69b9c779653947aca566a9f6ea7aca0b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/9065ae7439cd4742829a4e984f4765f1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/56ed001246b94ff789abefa79503a118, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/66e0cdcba4e04486b4fa01af298ec1a9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3eaf38e3c8f84c0ca445ebed6950b59e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d753a96a12ea43499c1a2cea7d65ea47, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/a60db7e6f3174ba8bbadedf9d0dfab90, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/95d8a99637714e2c9e561368ee4c518c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/933647ad9cea494b8a684ad4dbcca84c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/4e7245ae302c4b3a9b0b2cf121155381, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/11a658354a17442ca56355e54e932d11, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e20eb0cbf6e14055a4d69e0b90b239e0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3424337edbbc41da94354dee79f20a7c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ce2cc877b43a4e03a788d88dd73f463e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8f87704b24ba4e34a8ee8abf920a939b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/b3d6441f5afd48b7b759b5ec0fd8fd19, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/cfb5f6f3b3df47e3880f72d94206b37c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d7efd022e13f4bb8987be085e5f5ec70, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/913dbc2d6edb41198271faff880d730d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/184c135a71394f35a8f8a0969c13cf76, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c262ec71c87d4503adc57f4a6e8bd703, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/a3ea4ba7413444f79e7b7909ab7f802e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3fd17389fcb046ccaa4694db406dcabd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/6af6ebfc81bb46e5b76dcabe5fdd3fb1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/fb705c06a2154eef9bf5f224cd4d47a2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/1fa24aaba0494bbaa1972bdb831ac73c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e4c86246cccd4ebc8eb6c5d5c4455084, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ee06b3bb58e848c9a1f3ccc23c45e03a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c04eda19b7744c11a17ad3f7a331805b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c8690c6a0148460d9de77b239ea63b2b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/256486dc0ce04ee5bce04eb03b94bcf3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3815e1bdfc4f45faa0f3d165f1e8d08f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/76686f7beeb646348fe1f75fb8118cc8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8d6be7f1a61342a6a168f4cccd8c1a21, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/f9fd590eb6354002944f310707203e18, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/4d5487d7107f4c9293a4cd5e3e2687e6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e0c1ac77b6094047bd67140c0a6ebbd5] to archive 2023-04-03 02:56:17,227 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:56:17,228 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/32f84934fd0a4bd9a720c1ca4d2803cb to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/32f84934fd0a4bd9a720c1ca4d2803cb 2023-04-03 02:56:17,229 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/2434ab635f8045e292d93db2d16eca92 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/2434ab635f8045e292d93db2d16eca92 2023-04-03 02:56:17,231 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e076428bec3d4feb98c861427706bee2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e076428bec3d4feb98c861427706bee2 2023-04-03 02:56:17,232 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/84d5a088fbfd4d6e916beb9984407c0f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/84d5a088fbfd4d6e916beb9984407c0f 2023-04-03 02:56:17,233 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/69b9c779653947aca566a9f6ea7aca0b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/69b9c779653947aca566a9f6ea7aca0b 2023-04-03 02:56:17,234 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/9065ae7439cd4742829a4e984f4765f1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/9065ae7439cd4742829a4e984f4765f1 2023-04-03 02:56:17,235 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/56ed001246b94ff789abefa79503a118 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/56ed001246b94ff789abefa79503a118 2023-04-03 02:56:17,236 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/66e0cdcba4e04486b4fa01af298ec1a9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/66e0cdcba4e04486b4fa01af298ec1a9 2023-04-03 02:56:17,237 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3eaf38e3c8f84c0ca445ebed6950b59e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3eaf38e3c8f84c0ca445ebed6950b59e 2023-04-03 02:56:17,238 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d753a96a12ea43499c1a2cea7d65ea47 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d753a96a12ea43499c1a2cea7d65ea47 2023-04-03 02:56:17,239 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/a60db7e6f3174ba8bbadedf9d0dfab90 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/a60db7e6f3174ba8bbadedf9d0dfab90 2023-04-03 02:56:17,241 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/95d8a99637714e2c9e561368ee4c518c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/95d8a99637714e2c9e561368ee4c518c 2023-04-03 02:56:17,242 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/933647ad9cea494b8a684ad4dbcca84c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/933647ad9cea494b8a684ad4dbcca84c 2023-04-03 02:56:17,242 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/4e7245ae302c4b3a9b0b2cf121155381 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/4e7245ae302c4b3a9b0b2cf121155381 2023-04-03 02:56:17,244 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/11a658354a17442ca56355e54e932d11 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/11a658354a17442ca56355e54e932d11 2023-04-03 02:56:17,245 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e20eb0cbf6e14055a4d69e0b90b239e0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e20eb0cbf6e14055a4d69e0b90b239e0 2023-04-03 02:56:17,246 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3424337edbbc41da94354dee79f20a7c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3424337edbbc41da94354dee79f20a7c 2023-04-03 02:56:17,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ce2cc877b43a4e03a788d88dd73f463e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ce2cc877b43a4e03a788d88dd73f463e 2023-04-03 02:56:17,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8f87704b24ba4e34a8ee8abf920a939b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8f87704b24ba4e34a8ee8abf920a939b 2023-04-03 02:56:17,250 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/b3d6441f5afd48b7b759b5ec0fd8fd19 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/b3d6441f5afd48b7b759b5ec0fd8fd19 2023-04-03 02:56:17,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/cfb5f6f3b3df47e3880f72d94206b37c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/cfb5f6f3b3df47e3880f72d94206b37c 2023-04-03 02:56:17,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d7efd022e13f4bb8987be085e5f5ec70 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/d7efd022e13f4bb8987be085e5f5ec70 2023-04-03 02:56:17,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/913dbc2d6edb41198271faff880d730d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/913dbc2d6edb41198271faff880d730d 2023-04-03 02:56:17,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/184c135a71394f35a8f8a0969c13cf76 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/184c135a71394f35a8f8a0969c13cf76 2023-04-03 02:56:17,256 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c262ec71c87d4503adc57f4a6e8bd703 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c262ec71c87d4503adc57f4a6e8bd703 2023-04-03 02:56:17,257 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/a3ea4ba7413444f79e7b7909ab7f802e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/a3ea4ba7413444f79e7b7909ab7f802e 2023-04-03 02:56:17,258 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3fd17389fcb046ccaa4694db406dcabd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3fd17389fcb046ccaa4694db406dcabd 2023-04-03 02:56:17,258 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/6af6ebfc81bb46e5b76dcabe5fdd3fb1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/6af6ebfc81bb46e5b76dcabe5fdd3fb1 2023-04-03 02:56:17,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/fb705c06a2154eef9bf5f224cd4d47a2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/fb705c06a2154eef9bf5f224cd4d47a2 2023-04-03 02:56:17,260 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/1fa24aaba0494bbaa1972bdb831ac73c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/1fa24aaba0494bbaa1972bdb831ac73c 2023-04-03 02:56:17,261 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e4c86246cccd4ebc8eb6c5d5c4455084 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e4c86246cccd4ebc8eb6c5d5c4455084 2023-04-03 02:56:17,262 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ee06b3bb58e848c9a1f3ccc23c45e03a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/ee06b3bb58e848c9a1f3ccc23c45e03a 2023-04-03 02:56:17,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c04eda19b7744c11a17ad3f7a331805b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c04eda19b7744c11a17ad3f7a331805b 2023-04-03 02:56:17,264 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c8690c6a0148460d9de77b239ea63b2b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/c8690c6a0148460d9de77b239ea63b2b 2023-04-03 02:56:17,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/256486dc0ce04ee5bce04eb03b94bcf3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/256486dc0ce04ee5bce04eb03b94bcf3 2023-04-03 02:56:17,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3815e1bdfc4f45faa0f3d165f1e8d08f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/3815e1bdfc4f45faa0f3d165f1e8d08f 2023-04-03 02:56:17,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/76686f7beeb646348fe1f75fb8118cc8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/76686f7beeb646348fe1f75fb8118cc8 2023-04-03 02:56:17,267 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8d6be7f1a61342a6a168f4cccd8c1a21 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8d6be7f1a61342a6a168f4cccd8c1a21 2023-04-03 02:56:17,268 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/f9fd590eb6354002944f310707203e18 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/f9fd590eb6354002944f310707203e18 2023-04-03 02:56:17,269 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/4d5487d7107f4c9293a4cd5e3e2687e6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/4d5487d7107f4c9293a4cd5e3e2687e6 2023-04-03 02:56:17,270 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e0c1ac77b6094047bd67140c0a6ebbd5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/e0c1ac77b6094047bd67140c0a6ebbd5 2023-04-03 02:56:17,273 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/30182d05682641fd96bc74eadf0138cd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/332c3a8dfe564c1bb2a4aabd6f3ea666, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5a236f5134994231b940744b054ee034, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/9b596996541f47cfa7bc8077e6ca737f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/735326def0a34c3b8166a483bf79f95a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/175cff5813124af790de887ea4808388, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/defe6afc48b74d1e93aa0d9f5e528154, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/75b87767ab8a43e3bb1fb9269ce8dbc9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/c81884513b7d475b99ed96fabbf5d4c5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/9badc5da1e034e80898e6f6d5090966f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/904368f8af7749088daea9c265d2d6e0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b901c8b84b854acc88301279effbb54d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ddc2eb59d5114ec9ad16cd393cb97851, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4449ccfb94441fa9e2982ef3794a98, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1f4eca250cb042049440700d7943cd34, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/80894bd081374f12828ba6e3440a26b8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5991b3b8877c47f2b920166fffe25b27, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8ab35e192f4148c3a0fe153c1e503640, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/52c7cae976614ebda70dbf6ddf78745b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1a7dea6b390542f8a8e9e38d547a4945, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5dee288189434c8689dec302b2d08bf3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a66046a929dc4b12b01d65c340e326c5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/6545110422a245bb9e318d31f1a04c96, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/3e6ca87803f74dabab2fb08ab754de07, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2d085895b4d9458ba7c84068adb3368e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b8efbde2a943499b87f72e2dc73f6c70, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/99aabcbe6f764727b40fd409e6d350a9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/765db5a0117743daa07619eabaa2f788, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/cb27d39b936e4ce683a1c780d0f86ad8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4a5c1cd980429fbbbc2ea0a92a3cc1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/85fc66c9930e4f74a75784ddb268eb6d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/24cf6be1c2d140f68d7087fbeccff3a9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/e413a96b8b7d4d569831db30e23d0f06, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/76dea5f3642647b6a9bfb3f3e166c40a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/384b618dadde45b9b06f3d05786c45cc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ecdc985daee24368aa1781026ec88d0f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a21cf059664f4d5fa217a7eb33d0c2b2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8e1bacc1a2a4487fa48d43ae451e2830, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2aa1264af78d4cee871755325a0487f5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ddbd030993484c6197aa449b9036f960, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8f5742bd1af643aaa4f9e124a235ceb1] to archive 2023-04-03 02:56:17,274 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:56:17,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/30182d05682641fd96bc74eadf0138cd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/30182d05682641fd96bc74eadf0138cd 2023-04-03 02:56:17,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/332c3a8dfe564c1bb2a4aabd6f3ea666 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/332c3a8dfe564c1bb2a4aabd6f3ea666 2023-04-03 02:56:17,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5a236f5134994231b940744b054ee034 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5a236f5134994231b940744b054ee034 2023-04-03 02:56:17,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 02:56:17,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/9b596996541f47cfa7bc8077e6ca737f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/9b596996541f47cfa7bc8077e6ca737f 2023-04-03 02:56:17,278 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/735326def0a34c3b8166a483bf79f95a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/735326def0a34c3b8166a483bf79f95a 2023-04-03 02:56:17,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/175cff5813124af790de887ea4808388 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/175cff5813124af790de887ea4808388 2023-04-03 02:56:17,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/defe6afc48b74d1e93aa0d9f5e528154 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/defe6afc48b74d1e93aa0d9f5e528154 2023-04-03 02:56:17,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/75b87767ab8a43e3bb1fb9269ce8dbc9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/75b87767ab8a43e3bb1fb9269ce8dbc9 2023-04-03 02:56:17,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/c81884513b7d475b99ed96fabbf5d4c5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/c81884513b7d475b99ed96fabbf5d4c5 2023-04-03 02:56:17,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/9badc5da1e034e80898e6f6d5090966f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/9badc5da1e034e80898e6f6d5090966f 2023-04-03 02:56:17,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/904368f8af7749088daea9c265d2d6e0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/904368f8af7749088daea9c265d2d6e0 2023-04-03 02:56:17,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b901c8b84b854acc88301279effbb54d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b901c8b84b854acc88301279effbb54d 2023-04-03 02:56:17,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ddc2eb59d5114ec9ad16cd393cb97851 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ddc2eb59d5114ec9ad16cd393cb97851 2023-04-03 02:56:17,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4449ccfb94441fa9e2982ef3794a98 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4449ccfb94441fa9e2982ef3794a98 2023-04-03 02:56:17,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1f4eca250cb042049440700d7943cd34 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1f4eca250cb042049440700d7943cd34 2023-04-03 02:56:17,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/80894bd081374f12828ba6e3440a26b8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/80894bd081374f12828ba6e3440a26b8 2023-04-03 02:56:17,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5991b3b8877c47f2b920166fffe25b27 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5991b3b8877c47f2b920166fffe25b27 2023-04-03 02:56:17,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8ab35e192f4148c3a0fe153c1e503640 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8ab35e192f4148c3a0fe153c1e503640 2023-04-03 02:56:17,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/52c7cae976614ebda70dbf6ddf78745b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/52c7cae976614ebda70dbf6ddf78745b 2023-04-03 02:56:17,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1a7dea6b390542f8a8e9e38d547a4945 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1a7dea6b390542f8a8e9e38d547a4945 2023-04-03 02:56:17,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5dee288189434c8689dec302b2d08bf3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/5dee288189434c8689dec302b2d08bf3 2023-04-03 02:56:17,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a66046a929dc4b12b01d65c340e326c5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a66046a929dc4b12b01d65c340e326c5 2023-04-03 02:56:17,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/6545110422a245bb9e318d31f1a04c96 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/6545110422a245bb9e318d31f1a04c96 2023-04-03 02:56:17,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/3e6ca87803f74dabab2fb08ab754de07 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/3e6ca87803f74dabab2fb08ab754de07 2023-04-03 02:56:17,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2d085895b4d9458ba7c84068adb3368e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2d085895b4d9458ba7c84068adb3368e 2023-04-03 02:56:17,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b8efbde2a943499b87f72e2dc73f6c70 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/b8efbde2a943499b87f72e2dc73f6c70 2023-04-03 02:56:17,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/99aabcbe6f764727b40fd409e6d350a9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/99aabcbe6f764727b40fd409e6d350a9 2023-04-03 02:56:17,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/765db5a0117743daa07619eabaa2f788 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/765db5a0117743daa07619eabaa2f788 2023-04-03 02:56:17,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/cb27d39b936e4ce683a1c780d0f86ad8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/cb27d39b936e4ce683a1c780d0f86ad8 2023-04-03 02:56:17,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4a5c1cd980429fbbbc2ea0a92a3cc1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ec4a5c1cd980429fbbbc2ea0a92a3cc1 2023-04-03 02:56:17,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/85fc66c9930e4f74a75784ddb268eb6d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/85fc66c9930e4f74a75784ddb268eb6d 2023-04-03 02:56:17,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/24cf6be1c2d140f68d7087fbeccff3a9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/24cf6be1c2d140f68d7087fbeccff3a9 2023-04-03 02:56:17,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/e413a96b8b7d4d569831db30e23d0f06 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/e413a96b8b7d4d569831db30e23d0f06 2023-04-03 02:56:17,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/76dea5f3642647b6a9bfb3f3e166c40a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/76dea5f3642647b6a9bfb3f3e166c40a 2023-04-03 02:56:17,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/384b618dadde45b9b06f3d05786c45cc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/384b618dadde45b9b06f3d05786c45cc 2023-04-03 02:56:17,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ecdc985daee24368aa1781026ec88d0f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ecdc985daee24368aa1781026ec88d0f 2023-04-03 02:56:17,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a21cf059664f4d5fa217a7eb33d0c2b2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/a21cf059664f4d5fa217a7eb33d0c2b2 2023-04-03 02:56:17,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8e1bacc1a2a4487fa48d43ae451e2830 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8e1bacc1a2a4487fa48d43ae451e2830 2023-04-03 02:56:17,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2aa1264af78d4cee871755325a0487f5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/2aa1264af78d4cee871755325a0487f5 2023-04-03 02:56:17,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ddbd030993484c6197aa449b9036f960 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/ddbd030993484c6197aa449b9036f960 2023-04-03 02:56:17,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8f5742bd1af643aaa4f9e124a235ceb1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/8f5742bd1af643aaa4f9e124a235ceb1 2023-04-03 02:56:17,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b4169953bf7842c6bc437e7c16cde0bf, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/210366bd20cf44458b16f9977ab0a121, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a4b3f2e50a4042f2a9cff8d7674cd64d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/6ec0e6db198c41fe89165dffaec7aecb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/419f83b0638c4bfd97affa2f780ddd0f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2e81d33546b64baab3a6f50f4e70866e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/65872f7177d748deaf6f4317e4a0e1c9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c109cc8794e34f32bf8b08c902cd7794, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e3112f2ff1834332955369b7d28391f1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/8a10170a9b804e148f24b82f44accb79, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/07d110fe64ec428899ab55d1fc6d2243, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d309d8f3a1bf44de91bfcfc6521067a8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c8b01a2261ec4c2b879240b5b713d3bd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/054daed70ff044c299c47387203ac7d4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d06bfbe8bb054e62b0edf093fba3aeb1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/476aaf5a7a684c2ca4aeeefecd6f0158, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/f0057dfde4164aa5966be242656f0472, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/73b5d9374d8944d89f46f8f253db56c5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7dca4342f8924b849c3355691c4f3c02, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b95658d403d6447b918eb9c9e3ca6d73, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/0a2f1622062d437f95bedea824220929, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/eb25caccc949435ca06fddd335c6dcb7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2b59cb4da315453b98a4635aa26a7d8e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/9bfaf0f0e1034e2499c0eebfd1ace613, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c2df2e4768304a9aa443fe209ddf73d8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/f7683525288a4a419a4dda17360ff237, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a68614752a234ebeb797addbcbc015f3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c68f37a3982a4af3b456cb37503fbecc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/66a0ff0e6c204bfe88bdc3f013aca00e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/3abbd784683941c2a0c0ce20287c1c17, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/daf3e9e099244956a5e5e44a764af554, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/1396ba1f6d5840c2ac6c093fcaf88569, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/1065842b754741fdbeaf0216474bc2d3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/ca723af6af11463cb704ebbb1427103a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e3d8b0cc3e0948aaad251af1f5616e04, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7a75a785862a49e58ebaf2a1f633e73f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e898f2a5f5a341f48af83574d82896c5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/21e14d4852c44657ad19dd0e740d3af7, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/94f747a8199444dd915d7a9ee0c3f29a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a310cc300ac24c62ae09020475d82180] to archive 2023-04-03 02:56:17,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:56:17,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b4169953bf7842c6bc437e7c16cde0bf to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b4169953bf7842c6bc437e7c16cde0bf 2023-04-03 02:56:17,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/210366bd20cf44458b16f9977ab0a121 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/210366bd20cf44458b16f9977ab0a121 2023-04-03 02:56:17,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a4b3f2e50a4042f2a9cff8d7674cd64d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a4b3f2e50a4042f2a9cff8d7674cd64d 2023-04-03 02:56:17,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/6ec0e6db198c41fe89165dffaec7aecb to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/6ec0e6db198c41fe89165dffaec7aecb 2023-04-03 02:56:17,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/419f83b0638c4bfd97affa2f780ddd0f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/419f83b0638c4bfd97affa2f780ddd0f 2023-04-03 02:56:17,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2e81d33546b64baab3a6f50f4e70866e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2e81d33546b64baab3a6f50f4e70866e 2023-04-03 02:56:17,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/65872f7177d748deaf6f4317e4a0e1c9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/65872f7177d748deaf6f4317e4a0e1c9 2023-04-03 02:56:17,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c109cc8794e34f32bf8b08c902cd7794 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c109cc8794e34f32bf8b08c902cd7794 2023-04-03 02:56:17,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e3112f2ff1834332955369b7d28391f1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e3112f2ff1834332955369b7d28391f1 2023-04-03 02:56:17,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/8a10170a9b804e148f24b82f44accb79 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/8a10170a9b804e148f24b82f44accb79 2023-04-03 02:56:17,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/07d110fe64ec428899ab55d1fc6d2243 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/07d110fe64ec428899ab55d1fc6d2243 2023-04-03 02:56:17,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d309d8f3a1bf44de91bfcfc6521067a8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d309d8f3a1bf44de91bfcfc6521067a8 2023-04-03 02:56:17,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c8b01a2261ec4c2b879240b5b713d3bd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c8b01a2261ec4c2b879240b5b713d3bd 2023-04-03 02:56:17,318 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/054daed70ff044c299c47387203ac7d4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/054daed70ff044c299c47387203ac7d4 2023-04-03 02:56:17,319 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d06bfbe8bb054e62b0edf093fba3aeb1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/d06bfbe8bb054e62b0edf093fba3aeb1 2023-04-03 02:56:17,319 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/476aaf5a7a684c2ca4aeeefecd6f0158 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/476aaf5a7a684c2ca4aeeefecd6f0158 2023-04-03 02:56:17,320 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/f0057dfde4164aa5966be242656f0472 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/f0057dfde4164aa5966be242656f0472 2023-04-03 02:56:17,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/73b5d9374d8944d89f46f8f253db56c5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/73b5d9374d8944d89f46f8f253db56c5 2023-04-03 02:56:17,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7dca4342f8924b849c3355691c4f3c02 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7dca4342f8924b849c3355691c4f3c02 2023-04-03 02:56:17,322 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b95658d403d6447b918eb9c9e3ca6d73 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/b95658d403d6447b918eb9c9e3ca6d73 2023-04-03 02:56:17,322 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/0a2f1622062d437f95bedea824220929 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/0a2f1622062d437f95bedea824220929 2023-04-03 02:56:17,323 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/eb25caccc949435ca06fddd335c6dcb7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/eb25caccc949435ca06fddd335c6dcb7 2023-04-03 02:56:17,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2b59cb4da315453b98a4635aa26a7d8e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/2b59cb4da315453b98a4635aa26a7d8e 2023-04-03 02:56:17,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/9bfaf0f0e1034e2499c0eebfd1ace613 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/9bfaf0f0e1034e2499c0eebfd1ace613 2023-04-03 02:56:17,325 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c2df2e4768304a9aa443fe209ddf73d8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c2df2e4768304a9aa443fe209ddf73d8 2023-04-03 02:56:17,326 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/f7683525288a4a419a4dda17360ff237 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/f7683525288a4a419a4dda17360ff237 2023-04-03 02:56:17,326 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a68614752a234ebeb797addbcbc015f3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a68614752a234ebeb797addbcbc015f3 2023-04-03 02:56:17,327 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c68f37a3982a4af3b456cb37503fbecc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/c68f37a3982a4af3b456cb37503fbecc 2023-04-03 02:56:17,328 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/66a0ff0e6c204bfe88bdc3f013aca00e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/66a0ff0e6c204bfe88bdc3f013aca00e 2023-04-03 02:56:17,328 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/3abbd784683941c2a0c0ce20287c1c17 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/3abbd784683941c2a0c0ce20287c1c17 2023-04-03 02:56:17,329 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/daf3e9e099244956a5e5e44a764af554 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/daf3e9e099244956a5e5e44a764af554 2023-04-03 02:56:17,329 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/1396ba1f6d5840c2ac6c093fcaf88569 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/1396ba1f6d5840c2ac6c093fcaf88569 2023-04-03 02:56:17,330 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/1065842b754741fdbeaf0216474bc2d3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/1065842b754741fdbeaf0216474bc2d3 2023-04-03 02:56:17,331 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/ca723af6af11463cb704ebbb1427103a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/ca723af6af11463cb704ebbb1427103a 2023-04-03 02:56:17,331 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e3d8b0cc3e0948aaad251af1f5616e04 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e3d8b0cc3e0948aaad251af1f5616e04 2023-04-03 02:56:17,332 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7a75a785862a49e58ebaf2a1f633e73f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/7a75a785862a49e58ebaf2a1f633e73f 2023-04-03 02:56:17,333 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e898f2a5f5a341f48af83574d82896c5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/e898f2a5f5a341f48af83574d82896c5 2023-04-03 02:56:17,333 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/21e14d4852c44657ad19dd0e740d3af7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/21e14d4852c44657ad19dd0e740d3af7 2023-04-03 02:56:17,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/94f747a8199444dd915d7a9ee0c3f29a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/94f747a8199444dd915d7a9ee0c3f29a 2023-04-03 02:56:17,335 DEBUG [StoreCloser-TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a310cc300ac24c62ae09020475d82180 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/a310cc300ac24c62ae09020475d82180 2023-04-03 02:56:17,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/recovered.edits/1039.seqid, newMaxSeqId=1039, maxSeqId=1 2023-04-03 02:56:17,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5. 2023-04-03 02:56:17,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9abe279a88a55aefb769568e98ff0ef5: 2023-04-03 02:56:17,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:17,341 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=9abe279a88a55aefb769568e98ff0ef5, regionState=CLOSED 2023-04-03 02:56:17,342 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490577341"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490577341"}]},"ts":"1680490577341"} 2023-04-03 02:56:17,343 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-03 02:56:17,343 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 9abe279a88a55aefb769568e98ff0ef5, server=jenkins-hbase4.apache.org,43815,1680490421348 in 1.1610 sec 2023-04-03 02:56:17,344 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-03 02:56:17,345 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9abe279a88a55aefb769568e98ff0ef5, UNASSIGN in 1.1650 sec 2023-04-03 02:56:17,345 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490577345"}]},"ts":"1680490577345"} 2023-04-03 02:56:17,346 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 02:56:17,347 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 02:56:17,348 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1790 sec 2023-04-03 02:56:18,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 02:56:18,278 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-03 02:56:18,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 02:56:18,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:18,280 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:18,281 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:18,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-03 02:56:18,284 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:18,286 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/recovered.edits] 2023-04-03 02:56:18,288 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/703a89c4876e44a291ac087b5157db46 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/703a89c4876e44a291ac087b5157db46 2023-04-03 02:56:18,289 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/7c68f8abde3f4743a0cc66276aa265f7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/7c68f8abde3f4743a0cc66276aa265f7 2023-04-03 02:56:18,290 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8b22819905d3408ea2330dd8999aaecc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/8b22819905d3408ea2330dd8999aaecc 2023-04-03 02:56:18,290 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/f55c7c1bbdab4c538c1557730dbd948c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/A/f55c7c1bbdab4c538c1557730dbd948c 2023-04-03 02:56:18,292 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1e4dce8fad5a49beb02bb28aa9ad6e8d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/1e4dce8fad5a49beb02bb28aa9ad6e8d 2023-04-03 02:56:18,293 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/49e44cd405ad467cb3c16523dff1ebc9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/49e44cd405ad467cb3c16523dff1ebc9 2023-04-03 02:56:18,294 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/d0f7aaacfe0e4a0fad3e713c7ad70730 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/d0f7aaacfe0e4a0fad3e713c7ad70730 2023-04-03 02:56:18,295 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/eda4d61f774041baaa96c5e91a4e65f3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/B/eda4d61f774041baaa96c5e91a4e65f3 2023-04-03 02:56:18,297 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/32bfe153938945f4b5f48980db192279 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/32bfe153938945f4b5f48980db192279 2023-04-03 02:56:18,298 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/3cc3ae4ab3ba422b9455a052104b1e2b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/3cc3ae4ab3ba422b9455a052104b1e2b 2023-04-03 02:56:18,299 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/62a29c158c2048eaae49ff6553e37a74 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/62a29c158c2048eaae49ff6553e37a74 2023-04-03 02:56:18,299 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/ecc54fe6691c41f7a3a35fd931a508f9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/C/ecc54fe6691c41f7a3a35fd931a508f9 2023-04-03 02:56:18,301 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/recovered.edits/1039.seqid to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5/recovered.edits/1039.seqid 2023-04-03 02:56:18,302 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/9abe279a88a55aefb769568e98ff0ef5 2023-04-03 02:56:18,302 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:56:18,303 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:18,304 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 02:56:18,306 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 02:56:18,307 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:18,307 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 02:56:18,307 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680490578307"}]},"ts":"9223372036854775807"} 2023-04-03 02:56:18,308 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 02:56:18,308 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9abe279a88a55aefb769568e98ff0ef5, NAME => 'TestAcidGuarantees,,1680490553679.9abe279a88a55aefb769568e98ff0ef5.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 02:56:18,308 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 02:56:18,308 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680490578308"}]},"ts":"9223372036854775807"} 2023-04-03 02:56:18,309 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 02:56:18,311 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:18,312 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-03 02:56:18,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-03 02:56:18,382 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-03 02:56:18,399 INFO [Listener at localhost/46853] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=231 (was 226) - Thread LEAK? -, OpenFileDescriptor=563 (was 524) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=504 (was 440) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=3907 (was 3337) - AvailableMemoryMB LEAK? - 2023-04-03 02:56:18,414 INFO [Listener at localhost/46853] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=231, OpenFileDescriptor=563, MaxFileDescriptor=60000, SystemLoadAverage=504, ProcessCount=170, AvailableMemoryMB=3907 2023-04-03 02:56:18,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 02:56:18,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 02:56:18,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:18,419 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 02:56:18,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-03 02:56:18,419 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 02:56:18,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 02:56:18,420 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:18,421 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b empty. 2023-04-03 02:56:18,421 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:18,421 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:56:18,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 02:56:18,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 02:56:18,828 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 02:56:18,830 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1af30d70f4e72ebe50e43fde14e4fe9b, NAME => 'TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp 2023-04-03 02:56:18,841 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:56:18,842 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1af30d70f4e72ebe50e43fde14e4fe9b, disabling compactions & flushes 2023-04-03 02:56:18,842 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:18,842 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:18,842 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. after waiting 0 ms 2023-04-03 02:56:18,842 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:18,842 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:18,842 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:18,844 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 02:56:18,845 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490578845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490578845"}]},"ts":"1680490578845"} 2023-04-03 02:56:18,846 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 02:56:18,847 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 02:56:18,847 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490578847"}]},"ts":"1680490578847"} 2023-04-03 02:56:18,847 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 02:56:18,850 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1af30d70f4e72ebe50e43fde14e4fe9b, ASSIGN}] 2023-04-03 02:56:18,852 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1af30d70f4e72ebe50e43fde14e4fe9b, ASSIGN 2023-04-03 02:56:18,852 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1af30d70f4e72ebe50e43fde14e4fe9b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43815,1680490421348; forceNewPlan=false, retain=false 2023-04-03 02:56:19,003 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=1af30d70f4e72ebe50e43fde14e4fe9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:19,004 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490579003"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490579003"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490579003"}]},"ts":"1680490579003"} 2023-04-03 02:56:19,005 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:56:19,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 02:56:19,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:19,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1af30d70f4e72ebe50e43fde14e4fe9b, NAME => 'TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:56:19,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:19,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:56:19,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:19,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:19,163 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:19,164 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A 2023-04-03 02:56:19,164 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A 2023-04-03 02:56:19,164 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:56:19,165 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1af30d70f4e72ebe50e43fde14e4fe9b columnFamilyName A 2023-04-03 02:56:19,165 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(310): Store=1af30d70f4e72ebe50e43fde14e4fe9b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:56:19,165 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:19,166 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B 2023-04-03 02:56:19,166 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B 2023-04-03 02:56:19,166 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:56:19,167 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1af30d70f4e72ebe50e43fde14e4fe9b columnFamilyName B 2023-04-03 02:56:19,167 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(310): Store=1af30d70f4e72ebe50e43fde14e4fe9b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:56:19,167 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:19,169 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C 2023-04-03 02:56:19,169 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C 2023-04-03 02:56:19,169 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:56:19,169 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1af30d70f4e72ebe50e43fde14e4fe9b columnFamilyName C 2023-04-03 02:56:19,169 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(310): Store=1af30d70f4e72ebe50e43fde14e4fe9b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:56:19,169 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:19,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:19,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:19,172 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 02:56:19,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:19,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 02:56:19,176 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1af30d70f4e72ebe50e43fde14e4fe9b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60614265, jitterRate=-0.0967770665884018}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 02:56:19,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:19,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., pid=56, masterSystemTime=1680490579156 2023-04-03 02:56:19,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:19,178 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:19,178 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=1af30d70f4e72ebe50e43fde14e4fe9b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:19,179 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490579178"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680490579178"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680490579178"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680490579178"}]},"ts":"1680490579178"} 2023-04-03 02:56:19,181 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-03 02:56:19,181 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 in 174 msec 2023-04-03 02:56:19,182 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-03 02:56:19,182 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1af30d70f4e72ebe50e43fde14e4fe9b, ASSIGN in 331 msec 2023-04-03 02:56:19,183 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 02:56:19,183 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490579183"}]},"ts":"1680490579183"} 2023-04-03 02:56:19,184 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 02:56:19,187 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 02:56:19,187 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 771 msec 2023-04-03 02:56:19,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 02:56:19,523 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-03 02:56:19,523 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x03ced4ba to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:56:19,528 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45878790, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:56:19,529 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:56:19,530 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46048, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:56:19,531 DEBUG [Listener at localhost/46853] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 02:56:19,533 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48962, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 02:56:19,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 02:56:19,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 02:56:19,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:19,946 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 02:56:19,947 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 02:56:19,947 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 02:56:19,948 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-03 02:56:19,950 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1af30d70f4e72ebe50e43fde14e4fe9b, REOPEN/MOVE}] 2023-04-03 02:56:19,951 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1af30d70f4e72ebe50e43fde14e4fe9b, REOPEN/MOVE 2023-04-03 02:56:19,951 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1af30d70f4e72ebe50e43fde14e4fe9b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:19,951 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490579951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490579951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490579951"}]},"ts":"1680490579951"} 2023-04-03 02:56:19,952 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:56:20,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1af30d70f4e72ebe50e43fde14e4fe9b, disabling compactions & flushes 2023-04-03 02:56:20,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. after waiting 0 ms 2023-04-03 02:56:20,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-03 02:56:20,109 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:20,110 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 1af30d70f4e72ebe50e43fde14e4fe9b to self. 2023-04-03 02:56:20,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,111 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1af30d70f4e72ebe50e43fde14e4fe9b, regionState=CLOSED 2023-04-03 02:56:20,111 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490580111"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490580111"}]},"ts":"1680490580111"} 2023-04-03 02:56:20,113 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-03 02:56:20,113 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 in 160 msec 2023-04-03 02:56:20,113 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1af30d70f4e72ebe50e43fde14e4fe9b, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,43815,1680490421348; forceNewPlan=false, retain=true 2023-04-03 02:56:20,264 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1af30d70f4e72ebe50e43fde14e4fe9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,264 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490580264"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490580264"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490580264"}]},"ts":"1680490580264"} 2023-04-03 02:56:20,265 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:56:20,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1af30d70f4e72ebe50e43fde14e4fe9b, NAME => 'TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.', STARTKEY => '', ENDKEY => ''} 2023-04-03 02:56:20,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 02:56:20,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,422 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,423 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A 2023-04-03 02:56:20,423 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A 2023-04-03 02:56:20,423 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:56:20,424 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1af30d70f4e72ebe50e43fde14e4fe9b columnFamilyName A 2023-04-03 02:56:20,426 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(310): Store=1af30d70f4e72ebe50e43fde14e4fe9b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:56:20,427 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,427 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B 2023-04-03 02:56:20,427 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B 2023-04-03 02:56:20,428 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:56:20,428 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1af30d70f4e72ebe50e43fde14e4fe9b columnFamilyName B 2023-04-03 02:56:20,428 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(310): Store=1af30d70f4e72ebe50e43fde14e4fe9b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:56:20,428 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,429 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C 2023-04-03 02:56:20,429 DEBUG [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C 2023-04-03 02:56:20,429 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 02:56:20,429 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1af30d70f4e72ebe50e43fde14e4fe9b columnFamilyName C 2023-04-03 02:56:20,430 INFO [StoreOpener-1af30d70f4e72ebe50e43fde14e4fe9b-1] regionserver.HStore(310): Store=1af30d70f4e72ebe50e43fde14e4fe9b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 02:56:20,430 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,431 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,431 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 02:56:20,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1af30d70f4e72ebe50e43fde14e4fe9b; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66134770, jitterRate=-0.014515131711959839}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 02:56:20,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:20,436 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., pid=61, masterSystemTime=1680490580417 2023-04-03 02:56:20,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,437 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,438 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=1af30d70f4e72ebe50e43fde14e4fe9b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,438 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490580438"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680490580438"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680490580438"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680490580438"}]},"ts":"1680490580438"} 2023-04-03 02:56:20,440 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-03 02:56:20,440 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 in 174 msec 2023-04-03 02:56:20,441 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-03 02:56:20,441 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1af30d70f4e72ebe50e43fde14e4fe9b, REOPEN/MOVE in 490 msec 2023-04-03 02:56:20,444 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-03 02:56:20,444 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 494 msec 2023-04-03 02:56:20,446 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 909 msec 2023-04-03 02:56:20,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-03 02:56:20,448 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x63da10ca to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:56:20,458 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1644730f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:56:20,458 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x59185ba7 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:56:20,471 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6af74c0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:56:20,472 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x062b3caa to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:56:20,483 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d759aab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:56:20,483 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x66b5b1f3 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:56:20,492 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39241fe3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:56:20,492 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x544a4215 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:56:20,516 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1fbb5b46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:56:20,516 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:56:20,529 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:56:20,529 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x5e6a6f3c to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:56:20,545 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f4eab6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:56:20,545 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x0a470a3d to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:56:20,561 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d997c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:56:20,562 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x4bb72437 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:56:20,571 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7993d99e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:56:20,571 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(139): Connect 0x3dca23c0 to 127.0.0.1:52506 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 02:56:20,578 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fae3ed2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 02:56:20,598 DEBUG [hconnection-0x7dd06e1c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:56:20,601 DEBUG [hconnection-0x7f932a02-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:56:20,604 DEBUG [hconnection-0x6dd6c612-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:56:20,605 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46084, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:56:20,608 DEBUG [hconnection-0x1213ccf1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:56:20,611 DEBUG [hconnection-0x4fed9901-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:56:20,612 DEBUG [hconnection-0x599b44b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:56:20,612 DEBUG [hconnection-0x736f5d5b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:56:20,613 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46108, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:56:20,613 DEBUG [hconnection-0x74e96c9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:56:20,617 DEBUG [hconnection-0x3f9a48a0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:56:20,620 DEBUG [hconnection-0x195b7db9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 02:56:20,620 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46142, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:56:20,624 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:56:20,631 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:56:20,632 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:56:20,632 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:56:20,632 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46156, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:56:20,632 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46104, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:56:20,632 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46126, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 02:56:20,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:20,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:20,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:20,644 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:20,644 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:20,644 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:20,644 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:20,645 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,645 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,647 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,647 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:20,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:20,648 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,648 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:20,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:20,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:20,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:56:20,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:20,653 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:20,653 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,653 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:20,654 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:20,654 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:20,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:20,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:20,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:20,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:20,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:20,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:20,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:20,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:20,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:20,662 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,662 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:20,662 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,662 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:20,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:20,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c3f3360[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:20,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:20,664 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,665 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,665 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:20,665 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:20,665 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:20,665 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,665 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:20,666 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:20,667 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:20,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:20,669 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:20,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:20,669 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:20,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:20,670 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:20,670 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403369a4d6f9433405f8c65302362e16efa to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403369a4d6f9433405f8c65302362e16efa 2023-04-03 02:56:20,672 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,673 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:20,673 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:20,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:20,673 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490640673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:20,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,677 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490640674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490640674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490640675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:20,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490640675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,678 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/6b130abf926846d7a9caafcbecc47047 2023-04-03 02:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,679 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:20,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:20,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,680 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:20,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:20,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:20,680 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:20,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:20,680 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:20,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:20,680 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:20,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490640777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490640778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490640778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490640778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490640779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,781 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:20,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:20,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:20,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:20,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:20,784 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:20,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:20,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:20,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:20,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,786 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,786 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:20,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:20,786 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,786 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:20,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:20,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:20,787 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:20,787 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:20,787 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:20,787 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:20,787 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:20,787 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,787 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:20,788 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:20,788 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:20,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:20,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:20,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:20,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,789 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:20,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,789 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513682a1[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:20,791 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:20,791 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,793 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:20,793 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,794 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:20,795 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:20,795 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:20,795 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:20,795 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:20,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:20,797 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:20,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:20,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:20,797 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:20,797 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:20,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:20,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,799 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:20,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:20,800 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:20,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,801 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:20,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:20,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:20,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:20,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:20,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:20,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,803 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,803 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:20,803 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:20,804 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:20,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:20,904 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:20,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:20,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:20,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:20,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:20,909 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:20,909 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:20,909 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:20,909 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:20,909 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,909 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,911 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,912 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,912 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:20,912 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:20,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:20,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:20,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:20,915 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:20,916 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,916 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:20,916 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:20,916 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:20,916 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:20,919 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:20,919 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:20,919 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,919 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b974c16[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:20,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:20,921 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,922 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:20,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:20,922 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:20,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:20,924 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:20,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:20,925 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:20,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:20,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:20,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:20,928 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:20,928 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:20,929 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:20,929 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:20,929 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:20,931 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,931 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:20,931 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:20,932 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:20,933 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:20,935 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:20,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:20,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:20,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:20,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:20,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:20,935 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:20,936 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:20,936 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:20,936 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:20,936 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:20,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:20,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490640980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490640980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490640980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490640981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:20,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:20,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490640982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,036 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:21,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:21,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:21,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:21,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:21,040 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:21,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:21,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:21,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:21,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,040 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,042 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:21,042 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:21,042 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,042 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:21,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:21,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:21,043 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:21,043 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:21,043 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:21,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:21,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:21,044 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:21,044 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:21,044 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:21,044 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:21,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:21,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:21,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:21,044 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:21,046 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,046 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,046 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,046 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b981cc2[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:21,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:21,048 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,050 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:21,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:21,051 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:21,051 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,051 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:21,055 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:21,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:21,055 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:21,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:21,057 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:21,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:21,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:21,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:21,057 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:21,059 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,060 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,061 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,061 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:21,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,065 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:21,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:21,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:21,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:21,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,065 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,065 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:21,065 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:21,066 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:21,066 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:21,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:21,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:21,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:21,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/44fb0aae6e014b91a04d8ad46c05fae9 2023-04-03 02:56:21,165 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:21,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:21,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:21,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:21,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:21,169 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:21,169 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:21,169 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:21,169 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:21,170 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,170 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,172 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:21,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:21,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,172 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:21,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:21,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:21,174 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:21,174 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:21,174 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:21,174 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:21,174 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:21,174 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:21,174 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:21,174 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:21,175 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:21,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:21,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:21,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:21,175 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:21,176 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,176 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,177 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,177 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:21,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e499ce2[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:21,179 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:21,179 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,180 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,180 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:21,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,181 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:21,181 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:21,181 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:21,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:21,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:21,182 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:21,182 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:21,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:21,185 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:21,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:21,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:21,185 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:21,185 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:21,188 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,188 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:21,188 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,188 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,190 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:21,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:21,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:21,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,192 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:21,192 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:21,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:21,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:21,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:21,193 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:21,193 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:21,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490641281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490641282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490641284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490641284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490641285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,294 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:21,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:21,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:21,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:21,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:21,297 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:21,297 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:21,297 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:21,297 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:21,297 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,297 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,298 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:21,298 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,299 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,299 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:21,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:21,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:21,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:21,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:21,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:21,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:21,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:21,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:21,316 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:21,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:21,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:21,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:21,316 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:21,318 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,318 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:21,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0f2c00[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:21,320 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:21,320 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,322 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:21,322 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,322 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:21,324 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,324 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,324 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:21,324 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:21,326 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:21,327 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:21,327 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:21,327 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:21,327 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:21,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,328 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:21,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,329 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,331 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,334 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:21,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:21,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:21,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:21,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:21,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:21,335 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:21,335 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:21,334 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,335 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:21,435 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:21,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:21,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:21,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:21,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:21,439 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:21,439 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:21,439 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:21,439 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:21,443 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,443 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,446 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:21,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:21,446 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,447 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,447 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:21,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:21,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:21,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:21,448 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:21,448 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:21,448 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:21,460 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:21,460 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:21,460 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:21,460 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:21,460 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:21,463 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,463 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,465 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,465 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a59328b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:21,466 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:21,466 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,473 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,473 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,474 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:21,475 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:21,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:21,480 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:21,480 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:21,480 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:21,480 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:21,480 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:21,485 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,485 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:21,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,485 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:21,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:21,486 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,489 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,491 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:21,492 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:21,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:21,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:21,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:21,492 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:21,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:21,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:21,492 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:21,492 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:21,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:21,510 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/5abb02692aef454894da63309923768f 2023-04-03 02:56:21,515 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/6b130abf926846d7a9caafcbecc47047 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6b130abf926846d7a9caafcbecc47047 2023-04-03 02:56:21,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6b130abf926846d7a9caafcbecc47047, entries=150, sequenceid=17, filesize=25.1 K 2023-04-03 02:56:21,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/44fb0aae6e014b91a04d8ad46c05fae9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/44fb0aae6e014b91a04d8ad46c05fae9 2023-04-03 02:56:21,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/44fb0aae6e014b91a04d8ad46c05fae9, entries=150, sequenceid=17, filesize=11.6 K 2023-04-03 02:56:21,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/5abb02692aef454894da63309923768f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5abb02692aef454894da63309923768f 2023-04-03 02:56:21,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5abb02692aef454894da63309923768f, entries=150, sequenceid=17, filesize=11.6 K 2023-04-03 02:56:21,530 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1af30d70f4e72ebe50e43fde14e4fe9b in 877ms, sequenceid=17, compaction requested=false 2023-04-03 02:56:21,530 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:21,593 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:21,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:21,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:21,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:21,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:21,600 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:21,600 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:21,600 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:21,600 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:21,600 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,601 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,603 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:21,603 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,603 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,603 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:21,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:21,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:21,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:21,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:21,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:21,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:21,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:21,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:21,612 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:21,612 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:21,612 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:21,612 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 02:56:21,616 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:21,616 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:21,616 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:21,616 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:21,616 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:21,616 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:21,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:21,790 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:21,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490641799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490641800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490641800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490641804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490641806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490641905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490641905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490641906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490641908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:21,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:21,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490641909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:22,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490642108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:22,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490642108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:22,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490642109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:22,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490642111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:22,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490642111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,114 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403bee0e2eb5249444fba0005d5a59237b1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bee0e2eb5249444fba0005d5a59237b1 2023-04-03 02:56:22,118 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/f62d4c76399447d18b1d178f782125b5 2023-04-03 02:56:22,126 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/11c904578db74b468977d3629d8e1891 2023-04-03 02:56:22,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:22,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490642411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:22,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490642413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:22,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490642413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:22,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490642415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:22,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490642415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,535 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/57a6578d1ab94c3382e7b084936a7bcb 2023-04-03 02:56:22,540 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/f62d4c76399447d18b1d178f782125b5 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/f62d4c76399447d18b1d178f782125b5 2023-04-03 02:56:22,543 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/f62d4c76399447d18b1d178f782125b5, entries=150, sequenceid=36, filesize=25.1 K 2023-04-03 02:56:22,544 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/11c904578db74b468977d3629d8e1891 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/11c904578db74b468977d3629d8e1891 2023-04-03 02:56:22,548 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/11c904578db74b468977d3629d8e1891, entries=150, sequenceid=36, filesize=11.6 K 2023-04-03 02:56:22,549 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/57a6578d1ab94c3382e7b084936a7bcb as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/57a6578d1ab94c3382e7b084936a7bcb 2023-04-03 02:56:22,560 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/57a6578d1ab94c3382e7b084936a7bcb, entries=150, sequenceid=36, filesize=11.6 K 2023-04-03 02:56:22,561 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 1af30d70f4e72ebe50e43fde14e4fe9b in 949ms, sequenceid=36, compaction requested=false 2023-04-03 02:56:22,561 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:22,561 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:22,561 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:22,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:22,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:22,562 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:22,563 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:22,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:22,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,564 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:22,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:22,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715c410e[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:22,566 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:22,566 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,568 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,569 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:22,569 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:22,569 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:22,569 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,569 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:22,570 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,570 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:22,570 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:22,571 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:22,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:22,575 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:22,575 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:22,575 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:22,575 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:22,575 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:22,577 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,577 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:22,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,577 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:22,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,579 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,581 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,585 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:22,585 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:22,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:22,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:22,588 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,588 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,588 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,588 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,588 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,588 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,588 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,588 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:22,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:22,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:22,588 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:22,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:22,588 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:22,689 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:22,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:22,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:22,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:22,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:22,691 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:22,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:22,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:22,691 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:22,692 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,692 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,695 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:22,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:22,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,695 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:22,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:22,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:22,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:22,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:22,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:22,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:22,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:22,696 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:22,697 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:22,697 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 02:56:22,697 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:22,697 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:22,697 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:22,697 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:22,697 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:22,697 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:22,707 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040374ae96c252ae457dbf018cd2b5d89933 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040374ae96c252ae457dbf018cd2b5d89933 2023-04-03 02:56:22,713 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/230ab9197a0641acadff77913ad64600 2023-04-03 02:56:22,725 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/3f8525d4a53a4dd0b770b668f38a1df6 2023-04-03 02:56:22,739 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/467baf9bbf5b41f99885650f2665e720 2023-04-03 02:56:22,744 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/230ab9197a0641acadff77913ad64600 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/230ab9197a0641acadff77913ad64600 2023-04-03 02:56:22,748 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/230ab9197a0641acadff77913ad64600, entries=150, sequenceid=50, filesize=25.1 K 2023-04-03 02:56:22,749 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/3f8525d4a53a4dd0b770b668f38a1df6 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f8525d4a53a4dd0b770b668f38a1df6 2023-04-03 02:56:22,752 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f8525d4a53a4dd0b770b668f38a1df6, entries=150, sequenceid=50, filesize=11.6 K 2023-04-03 02:56:22,753 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/467baf9bbf5b41f99885650f2665e720 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/467baf9bbf5b41f99885650f2665e720 2023-04-03 02:56:22,764 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/467baf9bbf5b41f99885650f2665e720, entries=150, sequenceid=50, filesize=11.6 K 2023-04-03 02:56:22,765 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 1af30d70f4e72ebe50e43fde14e4fe9b in 68ms, sequenceid=50, compaction requested=true 2023-04-03 02:56:22,765 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:22,765 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:22,765 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:22,765 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:22,765 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,765 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:22,765 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:22,767 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,767 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:22,767 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,767 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:22,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:22,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73706d4b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:22,769 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:22,769 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,770 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,770 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:22,770 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:22,770 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:22,770 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,770 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:22,771 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,771 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:22,771 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:22,772 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:22,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:22,775 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:22,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:22,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:22,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:22,775 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:22,777 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,777 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:22,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,777 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:22,778 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,779 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,782 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:22,782 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:22,782 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,782 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,782 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,782 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:22,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:22,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:22,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:22,782 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:22,783 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:22,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:22,782 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,783 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,862 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 02:56:22,883 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:22,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:22,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:22,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:22,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:22,886 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:22,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:22,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:22,886 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:22,887 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,887 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,888 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:22,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:22,888 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,889 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,889 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:22,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:22,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:22,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:22,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:22,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:22,892 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:22,892 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:22,892 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:22,893 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:22,893 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:22,894 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:22,895 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,895 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:22,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:22,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21cd3009[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:22,906 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:22,906 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,908 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,908 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:22,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:22,909 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:22,909 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,909 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:22,910 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,910 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:22,910 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:22,910 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:22,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:22,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:22,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:22,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:22,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:22,920 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:22,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:22,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:22,920 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:22,920 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:22,922 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,922 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:22,922 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:22,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:22,923 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:22,924 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,931 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:22,931 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:22,931 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:22,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:22,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:22,931 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:22,931 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,932 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:22,932 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:22,932 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,932 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:22,932 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:22,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:22,932 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:22,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:22,932 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:22,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:22,967 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:56:22,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:22,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:22,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:22,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:22,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:22,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:23,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490582995, max timestamp=1680490583001Num uniques 859; 2023-04-03 02:56:23,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:56:23,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490582995, max timestamp=1680490583002Num uniques 859; 2023-04-03 02:56:23,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:56:23,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490582995, max timestamp=1680490583002Num uniques 859; 2023-04-03 02:56:23,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:56:23,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490643023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490643024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490643024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490643025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,032 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:23,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:23,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:23,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490643028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:23,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:23,056 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:23,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:23,056 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:23,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,057 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,058 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,058 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,058 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,059 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:23,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:23,060 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403ee8bc690afc64e20bdd8f2cf4616ae80 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ee8bc690afc64e20bdd8f2cf4616ae80 2023-04-03 02:56:23,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:23,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,069 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:23,084 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:23,084 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,084 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:23,084 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:23,084 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:23,084 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,084 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:23,084 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:23,084 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,089 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62c393d1[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:23,105 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:23,105 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,106 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:23,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:23,106 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:23,107 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,107 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:23,109 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,109 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:23,109 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,109 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:23,112 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:23,112 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:23,112 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:23,112 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:23,112 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:23,114 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,114 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,114 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,115 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,117 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,119 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:23,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:23,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:23,119 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,120 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:23,121 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:23,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490643132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490643132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490643133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490643133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490643139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,221 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:23,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:23,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:23,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:23,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:23,225 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:23,225 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,225 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:23,225 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:23,225 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,225 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,226 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,227 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,227 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,228 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:23,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:23,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:23,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:23,240 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,240 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:23,240 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:23,240 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:23,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:23,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:23,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:23,243 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:23,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7104eab6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:23,245 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:23,245 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,246 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,247 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:23,247 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:23,247 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:23,247 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,247 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:23,249 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:23,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:23,251 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:23,251 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:23,251 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:23,251 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:23,252 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:23,254 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,254 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,254 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,258 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:23,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:23,261 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,262 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,262 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,262 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,262 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,262 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:23,262 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:23,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490643337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490643337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490643339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490643339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490643341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,362 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:23,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:23,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:23,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:23,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:23,368 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:23,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:23,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:23,369 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,369 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,370 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,370 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,370 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:23,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:23,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:23,371 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:23,372 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,372 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:23,372 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:23,372 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:23,374 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:23,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ecbe8[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:23,376 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:23,376 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,379 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:23,380 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,380 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:23,382 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:23,382 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,382 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:23,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:23,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:23,386 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:23,386 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:23,386 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:23,386 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:23,386 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:23,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,388 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,388 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,388 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,389 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,391 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:23,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:23,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,392 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,392 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:23,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:23,392 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:23,492 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:23,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:23,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:23,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:23,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:23,496 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:23,496 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,496 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:23,496 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:23,496 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,496 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,501 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,501 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,501 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,501 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:23,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:23,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:23,503 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,503 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:23,503 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:23,503 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:23,505 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@496bfeb4[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:23,507 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:23,507 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,508 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:23,508 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,508 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:23,510 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,510 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:23,510 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,510 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:23,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:23,512 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:23,512 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:23,512 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:23,512 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:23,512 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:23,514 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,514 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,514 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,516 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:23,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:23,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:23,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,518 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:23,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,518 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:23,520 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/6f3058aaed284559b758f32100f34a8b 2023-04-03 02:56:23,619 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:23,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:23,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:23,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:23,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:23,622 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:23,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:23,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:23,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,624 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:23,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:23,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:23,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:23,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:23,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:23,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,626 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:23,626 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:23,626 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:23,626 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,626 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:23,626 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:23,626 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,626 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:23,626 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:23,626 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,626 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:23,626 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:23,628 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c8b153[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:23,629 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:23,629 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,632 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:23,632 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,632 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:23,634 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:23,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,634 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:23,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:23,637 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:23,637 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:23,637 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:23,637 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:23,637 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:23,638 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,638 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,639 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,639 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,640 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490643642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,642 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,643 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:23,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:23,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:23,643 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,643 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,643 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:23,643 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:23,643 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,643 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,643 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,643 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,643 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,643 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490643643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490643643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490643645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:23,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490643645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,743 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:23,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:23,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:23,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:23,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:23,746 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:23,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:23,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:23,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,749 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,749 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,749 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,749 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:23,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:23,750 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,750 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:23,750 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:23,750 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:23,751 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:23,751 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,751 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:23,751 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:23,752 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,752 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,752 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,753 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61cfac46[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:23,754 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:23,754 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,756 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,756 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:23,756 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:23,756 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,756 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:23,756 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:23,757 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,757 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:23,757 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,757 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:23,759 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:23,759 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:23,759 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:23,759 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:23,760 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:23,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,762 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,762 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,763 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,765 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,765 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,765 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:23,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:23,765 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,765 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,766 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:23,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:23,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,766 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:23,866 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:23,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:23,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:23,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:23,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:23,870 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:23,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:23,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:23,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,871 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,872 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,872 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,872 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,872 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:23,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:23,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:23,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:23,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:23,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:23,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:23,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:23,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:23,874 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,874 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:23,874 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:23,874 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:23,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:23,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:23,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:23,874 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:23,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,879 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:23,881 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16890a1c[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:23,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:23,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,885 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:23,885 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,885 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:23,886 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:23,891 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:23,891 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:23,891 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:23,891 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:23,891 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:23,893 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,893 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:23,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:23,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,893 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:23,894 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,896 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,899 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:23,899 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:23,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:23,899 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:23,899 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:23,899 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,899 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:23,899 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:23,899 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,900 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:23,900 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:23,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:23,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:23,900 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:23,900 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:23,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:23,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/ea781834d4764c6a9e778aa5b65a9929 2023-04-03 02:56:24,000 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:24,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:24,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:24,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:24,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:24,003 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:24,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:24,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:24,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:24,004 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,004 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,005 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:24,005 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:24,005 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,005 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:24,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:24,007 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:24,007 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:24,007 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:24,007 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:24,007 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:24,007 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:24,007 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,007 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:24,007 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:24,008 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:24,009 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,009 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:24,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:24,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d827[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:24,010 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:24,010 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,011 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,011 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:24,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:24,011 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:24,011 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:24,014 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:24,014 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:24,014 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:24,014 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:24,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:24,016 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:24,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:24,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:24,016 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:24,016 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:24,017 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,017 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:24,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,017 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:24,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:24,018 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,019 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,022 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:24,022 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,022 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:24,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:24,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:24,022 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,022 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,022 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,022 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,022 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:24,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:24,022 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:24,022 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:24,022 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:24,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:24,123 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:24,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:24,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:24,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:24,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:24,127 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:24,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:24,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:24,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:24,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,128 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:24,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,130 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,130 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:24,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:24,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:24,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:24,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:24,131 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:24,131 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:24,131 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:24,131 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:24,134 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:24,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:24,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:24,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:24,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72b1a35d[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:24,137 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:24,137 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,138 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:24,138 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:24,140 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:24,143 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:24,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:24,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:24,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:24,143 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:24,146 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,146 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:24,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,146 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:24,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:24,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:24,147 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490644144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,148 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:24,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490644147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:24,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:24,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490644147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:24,152 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:24,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:24,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:24,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:24,152 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:24,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:24,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:24,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:24,153 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:24,153 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:24,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:24,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490644152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490644150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,253 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:24,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:24,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:24,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:24,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:24,256 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:24,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:24,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:24,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:24,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,258 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:24,258 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,258 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,258 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:24,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:24,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:24,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:24,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:24,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:24,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:24,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:24,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:24,259 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:24,260 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:24,260 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:24,260 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:24,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,261 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:24,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:24,262 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,262 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:24,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:24,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e809cd1[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:24,263 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:24,263 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,264 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:24,264 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:24,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:24,264 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:24,269 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:24,272 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:24,272 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:24,272 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:24,272 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:24,272 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:24,274 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:24,274 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,274 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:24,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,280 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,283 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,283 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:24,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:24,284 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:24,284 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:24,283 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:24,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:24,284 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,284 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,284 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:24,284 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:24,284 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:24,284 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:24,340 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/216b9616d0e34e068792beb0bb35e512 2023-04-03 02:56:24,344 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/6f3058aaed284559b758f32100f34a8b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6f3058aaed284559b758f32100f34a8b 2023-04-03 02:56:24,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6f3058aaed284559b758f32100f34a8b, entries=400, sequenceid=62, filesize=59.0 K 2023-04-03 02:56:24,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/ea781834d4764c6a9e778aa5b65a9929 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ea781834d4764c6a9e778aa5b65a9929 2023-04-03 02:56:24,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ea781834d4764c6a9e778aa5b65a9929, entries=150, sequenceid=62, filesize=11.6 K 2023-04-03 02:56:24,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/216b9616d0e34e068792beb0bb35e512 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/216b9616d0e34e068792beb0bb35e512 2023-04-03 02:56:24,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/216b9616d0e34e068792beb0bb35e512, entries=150, sequenceid=62, filesize=11.6 K 2023-04-03 02:56:24,365 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1af30d70f4e72ebe50e43fde14e4fe9b in 1398ms, sequenceid=62, compaction requested=true 2023-04-03 02:56:24,365 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:24,365 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:24,365 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:24,365 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:24,365 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:24,366 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:24,369 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 137692 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:24,369 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:24,369 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/B is initiating minor compaction (all files) 2023-04-03 02:56:24,369 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/B in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:24,369 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/44fb0aae6e014b91a04d8ad46c05fae9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/11c904578db74b468977d3629d8e1891, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f8525d4a53a4dd0b770b668f38a1df6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ea781834d4764c6a9e778aa5b65a9929] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=46.4 K 2023-04-03 02:56:24,369 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/A is initiating minor compaction (all files) 2023-04-03 02:56:24,370 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/A in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:24,370 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6b130abf926846d7a9caafcbecc47047, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/f62d4c76399447d18b1d178f782125b5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/230ab9197a0641acadff77913ad64600, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6f3058aaed284559b758f32100f34a8b] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=134.5 K 2023-04-03 02:56:24,370 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 44fb0aae6e014b91a04d8ad46c05fae9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680490580643 2023-04-03 02:56:24,370 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6b130abf926846d7a9caafcbecc47047, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680490580643 2023-04-03 02:56:24,370 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting f62d4c76399447d18b1d178f782125b5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680490580672 2023-04-03 02:56:24,370 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 11c904578db74b468977d3629d8e1891, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680490580672 2023-04-03 02:56:24,371 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 230ab9197a0641acadff77913ad64600, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680490581798 2023-04-03 02:56:24,371 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 3f8525d4a53a4dd0b770b668f38a1df6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680490581798 2023-04-03 02:56:24,371 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6f3058aaed284559b758f32100f34a8b, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680490582953 2023-04-03 02:56:24,371 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ea781834d4764c6a9e778aa5b65a9929, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680490582953 2023-04-03 02:56:24,384 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:24,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:24,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:24,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:24,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:24,392 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:24,392 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:24,392 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:24,392 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:24,393 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,393 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,393 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#B#compaction#557 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:24,394 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,394 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:24,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:24,394 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:24,394 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:24,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:24,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:24,417 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:24,417 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:24,417 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:24,417 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-03 02:56:24,417 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:24,417 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:24,417 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:24,417 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:24,417 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:24,418 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:24,419 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#A#compaction#556 average throughput is 0.56 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:24,835 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/d0c1c43d29274430be664457fc1822b4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/d0c1c43d29274430be664457fc1822b4 2023-04-03 02:56:24,842 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/B of 1af30d70f4e72ebe50e43fde14e4fe9b into d0c1c43d29274430be664457fc1822b4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:24,842 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:24,842 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/B, priority=12, startTime=1680490584365; duration=0sec 2023-04-03 02:56:24,842 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:24,843 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:24,844 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:24,844 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/C is initiating minor compaction (all files) 2023-04-03 02:56:24,845 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/C in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:24,845 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5abb02692aef454894da63309923768f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/57a6578d1ab94c3382e7b084936a7bcb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/467baf9bbf5b41f99885650f2665e720, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/216b9616d0e34e068792beb0bb35e512] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=46.4 K 2023-04-03 02:56:24,845 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 5abb02692aef454894da63309923768f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680490580643 2023-04-03 02:56:24,846 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 57a6578d1ab94c3382e7b084936a7bcb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680490580672 2023-04-03 02:56:24,846 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 467baf9bbf5b41f99885650f2665e720, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680490581798 2023-04-03 02:56:24,847 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 216b9616d0e34e068792beb0bb35e512, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680490582953 2023-04-03 02:56:24,851 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403495cb8ced0724e5b86fd15ec1b154eb4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403495cb8ced0724e5b86fd15ec1b154eb4 2023-04-03 02:56:24,860 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#C#compaction#559 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:24,873 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/a0e09a82a89948d6b279768066016bd0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/a0e09a82a89948d6b279768066016bd0 2023-04-03 02:56:24,878 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/C of 1af30d70f4e72ebe50e43fde14e4fe9b into a0e09a82a89948d6b279768066016bd0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:24,878 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:24,878 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/C, priority=12, startTime=1680490584366; duration=0sec 2023-04-03 02:56:24,878 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:25,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:25,153 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:25,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490645162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490645162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490645164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490645165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490645163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,260 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/ed35b537b01946ecb11410e40e62f85a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/ed35b537b01946ecb11410e40e62f85a 2023-04-03 02:56:25,261 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/198846129a3549d79407310e81e61e92 2023-04-03 02:56:25,264 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/A of 1af30d70f4e72ebe50e43fde14e4fe9b into ed35b537b01946ecb11410e40e62f85a(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:25,264 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:25,264 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/A, priority=12, startTime=1680490584365; duration=0sec 2023-04-03 02:56:25,264 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:25,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490645266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490645270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490645270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490645270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490645270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,277 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/2bc840572eff4c2c82d930ddeb621514 2023-04-03 02:56:25,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490645469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490645472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490645472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490645472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490645473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,689 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/b61a7ce8608f407b962e8e18b2a8a9b9 2023-04-03 02:56:25,693 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/198846129a3549d79407310e81e61e92 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/198846129a3549d79407310e81e61e92 2023-04-03 02:56:25,697 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/198846129a3549d79407310e81e61e92, entries=150, sequenceid=98, filesize=25.1 K 2023-04-03 02:56:25,697 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/2bc840572eff4c2c82d930ddeb621514 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2bc840572eff4c2c82d930ddeb621514 2023-04-03 02:56:25,701 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2bc840572eff4c2c82d930ddeb621514, entries=150, sequenceid=98, filesize=11.6 K 2023-04-03 02:56:25,701 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/b61a7ce8608f407b962e8e18b2a8a9b9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b61a7ce8608f407b962e8e18b2a8a9b9 2023-04-03 02:56:25,704 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b61a7ce8608f407b962e8e18b2a8a9b9, entries=150, sequenceid=98, filesize=11.6 K 2023-04-03 02:56:25,705 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 1af30d70f4e72ebe50e43fde14e4fe9b in 1288ms, sequenceid=98, compaction requested=false 2023-04-03 02:56:25,705 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:25,705 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:25,707 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,707 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:25,708 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:25,708 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3087b2f7[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:25,709 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:25,710 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,711 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,711 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:25,711 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,711 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:25,711 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:25,711 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:25,712 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:25,712 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:25,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:25,715 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:25,715 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:25,715 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:25,715 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:25,715 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:25,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:25,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:25,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:25,717 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,719 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,722 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:25,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:25,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:25,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:25,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:25,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:25,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:25,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,722 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:25,722 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:25,722 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:25,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:25,775 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 02:56:25,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:25,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:25,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:25,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:25,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:25,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:25,786 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040306927f55d7fb46648a0d560c1772d5f5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040306927f55d7fb46648a0d560c1772d5f5 2023-04-03 02:56:25,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490645789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490645790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490645791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490645792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490645791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,822 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:25,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:25,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:25,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:25,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:25,826 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:25,826 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:25,826 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:25,826 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:25,826 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,826 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,827 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,827 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:25,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:25,828 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,828 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:25,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:25,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:25,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:25,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:25,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:25,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:25,829 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:25,830 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:25,830 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:25,830 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:25,830 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:25,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:25,832 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,832 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:25,833 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:25,833 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:25,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:25,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec99420[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:25,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:25,834 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,835 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:25,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:25,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:25,837 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:25,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:25,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:25,840 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:25,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:25,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:25,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:25,841 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:25,842 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,842 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:25,842 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:25,843 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,845 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,848 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,848 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,848 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,848 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,848 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,848 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,848 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:25,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:25,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:25,848 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:25,848 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:25,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:25,848 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:25,848 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:25,848 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:25,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:25,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490645893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490645895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490645895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490645896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:25,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490645896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,948 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:25,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:25,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:25,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:25,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:25,952 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:25,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:25,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:25,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:25,953 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,953 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,954 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:25,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:25,954 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,955 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,955 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:25,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:25,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:25,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:25,956 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:25,956 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:25,956 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:25,960 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:25,960 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:25,960 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:25,960 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:25,960 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:25,964 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:25,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:25,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,964 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:25,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:25,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@419d63ef[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:25,967 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:25,967 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,968 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,968 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:25,968 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:25,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:25,969 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,969 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:25,971 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:25,971 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:25,971 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:25,971 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:25,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:25,975 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:25,975 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:25,975 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:25,975 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:25,975 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:25,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:25,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:25,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,983 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:25,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:25,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,984 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,985 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,988 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:25,988 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:25,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:25,989 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,989 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,989 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:25,989 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:25,989 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,989 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:25,989 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:25,989 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:25,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:25,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:25,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:25,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:25,989 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:25,989 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:26,090 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:26,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:26,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:26,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:26,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:26,093 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:26,093 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:26,094 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:26,094 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:26,094 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,094 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,095 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:26,095 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,096 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,097 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:26,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:26,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490646096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:26,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:26,098 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:26,098 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:26,099 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:26,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:26,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:26,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490646099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,101 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:26,101 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:26,101 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:26,101 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:26,101 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:26,101 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:26,101 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,101 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:26,101 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:26,103 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:26,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490646101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,103 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:26,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490646103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490646103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,105 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,105 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:26,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:26,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b111919[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:26,107 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:26,107 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,110 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:26,111 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:26,111 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:26,111 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:26,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:26,113 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:26,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:26,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:26,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:26,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:26,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:26,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:26,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:26,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:26,118 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:26,118 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:26,118 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:26,119 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:26,119 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:26,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:26,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,121 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:26,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:26,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,126 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:26,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:26,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:26,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:26,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:26,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:26,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:26,127 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,128 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,128 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,128 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,128 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:26,128 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:26,190 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=114, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/e0d709d8a44e476bb4d1ed9ff2f5048a 2023-04-03 02:56:26,205 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/e5e9887304414fea962a7dae11ebbe4d 2023-04-03 02:56:26,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/648a248e37934103bf18ee14287370ab 2023-04-03 02:56:26,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/e0d709d8a44e476bb4d1ed9ff2f5048a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/e0d709d8a44e476bb4d1ed9ff2f5048a 2023-04-03 02:56:26,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/e0d709d8a44e476bb4d1ed9ff2f5048a, entries=200, sequenceid=114, filesize=31.9 K 2023-04-03 02:56:26,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/e5e9887304414fea962a7dae11ebbe4d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e5e9887304414fea962a7dae11ebbe4d 2023-04-03 02:56:26,224 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e5e9887304414fea962a7dae11ebbe4d, entries=150, sequenceid=114, filesize=11.6 K 2023-04-03 02:56:26,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/648a248e37934103bf18ee14287370ab as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/648a248e37934103bf18ee14287370ab 2023-04-03 02:56:26,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/648a248e37934103bf18ee14287370ab, entries=150, sequenceid=114, filesize=11.6 K 2023-04-03 02:56:26,228 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:26,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:26,229 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 1af30d70f4e72ebe50e43fde14e4fe9b in 454ms, sequenceid=114, compaction requested=true 2023-04-03 02:56:26,229 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:26,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:26,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:26,229 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:26,229 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:26,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:26,230 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84316 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:26,230 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:26,230 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/A is initiating minor compaction (all files) 2023-04-03 02:56:26,230 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/B is initiating minor compaction (all files) 2023-04-03 02:56:26,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:26,230 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/B in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:26,230 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/A in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:26,230 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/d0c1c43d29274430be664457fc1822b4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2bc840572eff4c2c82d930ddeb621514, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e5e9887304414fea962a7dae11ebbe4d] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=35.0 K 2023-04-03 02:56:26,230 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/ed35b537b01946ecb11410e40e62f85a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/198846129a3549d79407310e81e61e92, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/e0d709d8a44e476bb4d1ed9ff2f5048a] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=82.3 K 2023-04-03 02:56:26,231 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting d0c1c43d29274430be664457fc1822b4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680490582953 2023-04-03 02:56:26,231 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting ed35b537b01946ecb11410e40e62f85a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680490582953 2023-04-03 02:56:26,231 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 2bc840572eff4c2c82d930ddeb621514, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680490583012 2023-04-03 02:56:26,231 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 198846129a3549d79407310e81e61e92, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680490583012 2023-04-03 02:56:26,231 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e5e9887304414fea962a7dae11ebbe4d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680490585163 2023-04-03 02:56:26,231 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting e0d709d8a44e476bb4d1ed9ff2f5048a, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680490585162 2023-04-03 02:56:26,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:26,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:26,232 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:26,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:26,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:26,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:26,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,235 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,235 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:26,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:26,235 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,235 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:26,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:26,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:26,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:26,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:26,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:26,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:26,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:26,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:26,236 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:26,236 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:26,236 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 02:56:26,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:26,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:26,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:26,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:26,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:26,237 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:26,245 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#B#compaction#566 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:26,255 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#A#compaction#565 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:26,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:26,401 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:26,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490646408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490646409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490646433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490646433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490646433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490646534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490646534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490646537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490646537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490646537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,661 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/b0da13f0c0eb48c695512326510d0901 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/b0da13f0c0eb48c695512326510d0901 2023-04-03 02:56:26,665 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040399e3ffb8c415468c810a105edf90c849 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040399e3ffb8c415468c810a105edf90c849 2023-04-03 02:56:26,666 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/B of 1af30d70f4e72ebe50e43fde14e4fe9b into b0da13f0c0eb48c695512326510d0901(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:26,667 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:26,667 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/B, priority=13, startTime=1680490586229; duration=0sec 2023-04-03 02:56:26,667 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:26,667 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:26,669 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:26,669 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/C is initiating minor compaction (all files) 2023-04-03 02:56:26,669 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/C in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:26,669 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/a0e09a82a89948d6b279768066016bd0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b61a7ce8608f407b962e8e18b2a8a9b9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/648a248e37934103bf18ee14287370ab] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=35.0 K 2023-04-03 02:56:26,670 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a0e09a82a89948d6b279768066016bd0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680490582953 2023-04-03 02:56:26,670 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b61a7ce8608f407b962e8e18b2a8a9b9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680490583012 2023-04-03 02:56:26,670 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 648a248e37934103bf18ee14287370ab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680490585163 2023-04-03 02:56:26,672 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/5cbb43b2787b4e4c9d53298c6c7bcb4c 2023-04-03 02:56:26,675 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#C#compaction#568 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:26,691 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/e206d467bd66418a8e1f61ec81965c76 2023-04-03 02:56:26,693 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/4227b0172b9b4bd28ab844befe445bfb as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/4227b0172b9b4bd28ab844befe445bfb 2023-04-03 02:56:26,704 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/C of 1af30d70f4e72ebe50e43fde14e4fe9b into 4227b0172b9b4bd28ab844befe445bfb(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:26,704 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:26,704 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/C, priority=13, startTime=1680490586229; duration=0sec 2023-04-03 02:56:26,704 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:26,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490646736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490646737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,742 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/92437813621447509aa7b129cebef9b3 2023-04-03 02:56:26,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490646740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490646740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:26,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490646740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,746 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/5cbb43b2787b4e4c9d53298c6c7bcb4c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5cbb43b2787b4e4c9d53298c6c7bcb4c 2023-04-03 02:56:26,750 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5cbb43b2787b4e4c9d53298c6c7bcb4c, entries=150, sequenceid=134, filesize=25.2 K 2023-04-03 02:56:26,751 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/e206d467bd66418a8e1f61ec81965c76 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e206d467bd66418a8e1f61ec81965c76 2023-04-03 02:56:26,754 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e206d467bd66418a8e1f61ec81965c76, entries=150, sequenceid=134, filesize=11.7 K 2023-04-03 02:56:26,755 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/92437813621447509aa7b129cebef9b3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/92437813621447509aa7b129cebef9b3 2023-04-03 02:56:26,758 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/92437813621447509aa7b129cebef9b3, entries=150, sequenceid=134, filesize=11.7 K 2023-04-03 02:56:26,759 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1af30d70f4e72ebe50e43fde14e4fe9b in 523ms, sequenceid=134, compaction requested=false 2023-04-03 02:56:26,759 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:26,759 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:26,759 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:26,759 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:26,759 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,759 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:26,759 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,760 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:26,761 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,761 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:26,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:26,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@260c1686[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:26,763 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:26,763 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,765 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,765 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,765 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:26,765 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:26,765 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:26,765 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:26,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:26,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:26,770 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:26,770 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:26,770 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:26,770 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:26,770 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:26,772 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,772 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:26,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,772 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:26,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:26,773 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:26,774 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:26,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:26,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:26,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,776 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:26,777 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:26,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:26,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:26,777 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:26,777 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:26,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:26,877 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:26,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:26,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:26,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:26,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:26,881 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:26,881 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:26,881 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:26,881 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:26,881 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,881 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,882 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:26,882 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:26,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:26,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,884 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:26,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:26,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:26,884 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:26,884 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:26,884 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:26,884 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:26,884 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:26,884 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:26,885 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:26,885 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:56:26,885 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:26,885 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:26,885 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:26,885 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:26,885 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:26,885 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:26,906 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403f1d335738d6b4e2aaca484c9dc96d25e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f1d335738d6b4e2aaca484c9dc96d25e 2023-04-03 02:56:26,912 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=148, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/851ac621215847e1a5f2c4337aeaa5f8 2023-04-03 02:56:26,930 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/3f07b41fd7604e7ebe584eafb1c9bd86 2023-04-03 02:56:26,965 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/1856f681960f43d0811c30b35fa56d0f 2023-04-03 02:56:26,969 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/851ac621215847e1a5f2c4337aeaa5f8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/851ac621215847e1a5f2c4337aeaa5f8 2023-04-03 02:56:26,974 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/851ac621215847e1a5f2c4337aeaa5f8, entries=150, sequenceid=148, filesize=25.3 K 2023-04-03 02:56:26,974 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/3f07b41fd7604e7ebe584eafb1c9bd86 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f07b41fd7604e7ebe584eafb1c9bd86 2023-04-03 02:56:26,977 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f07b41fd7604e7ebe584eafb1c9bd86, entries=150, sequenceid=148, filesize=11.8 K 2023-04-03 02:56:26,978 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/1856f681960f43d0811c30b35fa56d0f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/1856f681960f43d0811c30b35fa56d0f 2023-04-03 02:56:26,994 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/1856f681960f43d0811c30b35fa56d0f, entries=150, sequenceid=148, filesize=11.8 K 2023-04-03 02:56:26,994 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 1af30d70f4e72ebe50e43fde14e4fe9b in 109ms, sequenceid=148, compaction requested=true 2023-04-03 02:56:26,995 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:26,995 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,997 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5496fef3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:27,000 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:27,000 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,002 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:27,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,003 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:27,004 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,004 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,004 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,004 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:27,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:27,007 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:27,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:27,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:27,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:27,007 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:27,009 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,009 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,009 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,009 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,010 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,018 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,018 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,019 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,018 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:27,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:27,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:27,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,019 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:27,019 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:27,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:27,061 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:56:27,061 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:27,061 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:27,061 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:27,061 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:27,061 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:27,061 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:27,072 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304038c97a356dea441269eacbb7241b1722c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038c97a356dea441269eacbb7241b1722c 2023-04-03 02:56:27,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490587070, max timestamp=1680490587078Num uniques 859; 2023-04-03 02:56:27,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:56:27,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490587070, max timestamp=1680490587078Num uniques 859; 2023-04-03 02:56:27,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:56:27,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490587070, max timestamp=1680490587078Num uniques 859; 2023-04-03 02:56:27,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:56:27,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490647092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490647093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490587091, max timestamp=1680490587098Num uniques 1018; 2023-04-03 02:56:27,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 02:56:27,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490587091, max timestamp=1680490587098Num uniques 1018; 2023-04-03 02:56:27,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 02:56:27,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680490587091, max timestamp=1680490587098Num uniques 1018; 2023-04-03 02:56:27,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 02:56:27,113 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/d83b2839ec5644658934155e051a2346 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/d83b2839ec5644658934155e051a2346 2023-04-03 02:56:27,117 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/A of 1af30d70f4e72ebe50e43fde14e4fe9b into d83b2839ec5644658934155e051a2346(size=25.4 K), total size for store is 75.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:27,117 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:27,117 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/A, priority=13, startTime=1680490586229; duration=0sec 2023-04-03 02:56:27,117 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:27,119 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:27,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:27,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:27,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:27,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:27,123 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:27,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:27,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:27,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,124 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:27,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:27,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:27,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:27,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:27,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:27,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:27,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:27,126 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,126 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:27,126 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:27,126 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,126 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:27,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:27,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:27,127 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:27,128 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,129 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,129 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ebe6d95[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:27,130 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:27,130 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490647127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490647127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490647129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,132 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,132 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,132 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:27,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:27,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:27,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:27,134 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:27,136 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:27,137 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:27,137 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:27,137 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:27,137 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:27,138 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,138 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,138 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,140 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,142 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:27,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:27,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:27,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,142 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,143 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:27,143 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:27,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490647201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490647201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490647231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490647231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490647232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,243 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:27,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:27,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:27,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:27,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:27,257 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:27,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:27,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:27,258 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,258 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,261 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,261 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,261 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:27,262 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,262 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:27,262 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:27,262 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:27,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:27,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:27,264 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,264 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:27,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d447a8a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:27,266 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:27,266 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:27,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:27,268 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,268 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:27,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:27,269 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,269 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:27,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:27,272 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:27,272 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:27,272 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:27,272 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:27,272 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:27,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,276 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,279 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:27,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:27,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,280 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:27,280 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:27,380 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:27,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:27,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:27,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:27,383 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:27,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:27,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:27,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,387 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:27,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:27,389 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,389 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:27,389 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:27,389 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:27,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,393 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,393 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b3aff82[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:27,394 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:27,394 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,399 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:27,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:27,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:27,399 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,399 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:27,403 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,403 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:27,403 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,403 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:27,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:27,406 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:27,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:27,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:27,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:27,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,406 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:27,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490647404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490647404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,409 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,410 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,412 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,424 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,424 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,424 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,424 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:27,424 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:27,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:27,424 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,424 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,424 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,424 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,424 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,424 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:27,424 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490647434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490647434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490647435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,477 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/5362183ac0894f73b80b8e8ceb18e3ac 2023-04-03 02:56:27,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/7b103572272f47ae8b2212962783e762 2023-04-03 02:56:27,525 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:27,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:27,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:27,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:27,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:27,527 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:27,527 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,527 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:27,528 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:27,528 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,528 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,529 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,529 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,530 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:27,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:27,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:27,531 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,531 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:27,531 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:27,531 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:27,533 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,533 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:27,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78b8fb70[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:27,534 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:27,535 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,537 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,537 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:27,537 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:27,537 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,537 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:27,537 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:27,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:27,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,539 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:27,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:27,541 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:27,541 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:27,541 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:27,541 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:27,541 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:27,542 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,542 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,542 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,543 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,544 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,547 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:27,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:27,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,547 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:27,547 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,547 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:27,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:27,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,648 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:27,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:27,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:27,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:27,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:27,651 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:27,651 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,651 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:27,651 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:27,651 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,651 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,653 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,653 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:27,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:27,654 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:27,655 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,655 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:27,655 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:27,655 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:27,657 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,657 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7567f337[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:27,658 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:27,658 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,659 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,660 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:27,660 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,660 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:27,660 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:27,660 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:27,661 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,661 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:27,661 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,661 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:27,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:27,663 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:27,663 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:27,663 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:27,663 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:27,663 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:27,666 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,666 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,666 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,666 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,667 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:27,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:27,670 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,670 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:27,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,670 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:27,670 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490647707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490647709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490647737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490647737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:27,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490647738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,770 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:27,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:27,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:27,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:27,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:27,775 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:27,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:27,775 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:27,776 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,776 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,777 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,777 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,778 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,778 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:27,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:27,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:27,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:27,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,780 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:27,788 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,788 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:27,788 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:27,788 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:27,790 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,790 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,791 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,791 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:27,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b09573b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:27,792 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:27,793 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,794 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,794 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:27,794 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:27,794 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:27,795 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,795 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:27,797 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:27,797 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,797 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:27,797 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:27,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:27,799 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:27,799 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:27,799 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:27,799 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:27,800 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:27,801 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,801 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,801 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:27,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:27,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,803 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,805 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,808 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,808 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,808 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:27,808 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,808 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:27,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:27,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:27,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:27,808 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:27,809 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:27,809 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:27,809 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,809 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,809 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:27,809 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,893 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/e260f62d87df4e119066897a7c25673b 2023-04-03 02:56:27,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/5362183ac0894f73b80b8e8ceb18e3ac as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5362183ac0894f73b80b8e8ceb18e3ac 2023-04-03 02:56:27,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5362183ac0894f73b80b8e8ceb18e3ac, entries=200, sequenceid=158, filesize=32.1 K 2023-04-03 02:56:27,902 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/7b103572272f47ae8b2212962783e762 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/7b103572272f47ae8b2212962783e762 2023-04-03 02:56:27,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/7b103572272f47ae8b2212962783e762, entries=150, sequenceid=158, filesize=11.8 K 2023-04-03 02:56:27,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/e260f62d87df4e119066897a7c25673b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/e260f62d87df4e119066897a7c25673b 2023-04-03 02:56:27,909 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:27,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:27,910 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/e260f62d87df4e119066897a7c25673b, entries=150, sequenceid=158, filesize=11.8 K 2023-04-03 02:56:27,910 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1af30d70f4e72ebe50e43fde14e4fe9b in 849ms, sequenceid=158, compaction requested=true 2023-04-03 02:56:27,910 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:27,911 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:27,911 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:27,911 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:27,911 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:27,911 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:27,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:27,912 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110614 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:27,912 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/A is initiating minor compaction (all files) 2023-04-03 02:56:27,912 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/A in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,913 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48184 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:27,913 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/d83b2839ec5644658934155e051a2346, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5cbb43b2787b4e4c9d53298c6c7bcb4c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/851ac621215847e1a5f2c4337aeaa5f8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5362183ac0894f73b80b8e8ceb18e3ac] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=108.0 K 2023-04-03 02:56:27,913 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/B is initiating minor compaction (all files) 2023-04-03 02:56:27,913 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/B in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,913 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/b0da13f0c0eb48c695512326510d0901, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e206d467bd66418a8e1f61ec81965c76, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f07b41fd7604e7ebe584eafb1c9bd86, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/7b103572272f47ae8b2212962783e762] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=47.1 K 2023-04-03 02:56:27,913 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting d83b2839ec5644658934155e051a2346, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680490585163 2023-04-03 02:56:27,913 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b0da13f0c0eb48c695512326510d0901, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680490585163 2023-04-03 02:56:27,914 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e206d467bd66418a8e1f61ec81965c76, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680490585784 2023-04-03 02:56:27,914 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 5cbb43b2787b4e4c9d53298c6c7bcb4c, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680490585784 2023-04-03 02:56:27,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:27,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:27,914 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:27,914 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,915 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:27,915 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:27,915 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 3f07b41fd7604e7ebe584eafb1c9bd86, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680490586408 2023-04-03 02:56:27,915 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 851ac621215847e1a5f2c4337aeaa5f8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680490586408 2023-04-03 02:56:27,915 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 5362183ac0894f73b80b8e8ceb18e3ac, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680490587058 2023-04-03 02:56:27,916 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,916 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,916 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7b103572272f47ae8b2212962783e762, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680490587058 2023-04-03 02:56:27,918 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:27,918 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:27,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:27,919 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:27,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:27,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:27,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:27,924 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:27,924 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,924 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:27,924 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=134.18 KB heapSize=384.14 KB 2023-04-03 02:56:27,924 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:27,924 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:27,924 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:27,925 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:27,925 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:27,925 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:27,932 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#B#compaction#578 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:27,934 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#A#compaction#577 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:27,955 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/58ce3d87840949f481b6df724f8db2f4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/58ce3d87840949f481b6df724f8db2f4 2023-04-03 02:56:27,959 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/B of 1af30d70f4e72ebe50e43fde14e4fe9b into 58ce3d87840949f481b6df724f8db2f4(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:27,959 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:27,959 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/B, priority=12, startTime=1680490587911; duration=0sec 2023-04-03 02:56:27,959 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:27,959 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:27,960 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48184 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:27,960 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/C is initiating minor compaction (all files) 2023-04-03 02:56:27,960 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/C in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:27,960 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/4227b0172b9b4bd28ab844befe445bfb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/92437813621447509aa7b129cebef9b3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/1856f681960f43d0811c30b35fa56d0f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/e260f62d87df4e119066897a7c25673b] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=47.1 K 2023-04-03 02:56:27,961 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 4227b0172b9b4bd28ab844befe445bfb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1680490585163 2023-04-03 02:56:27,961 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 92437813621447509aa7b129cebef9b3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680490585784 2023-04-03 02:56:27,961 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 1856f681960f43d0811c30b35fa56d0f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680490586408 2023-04-03 02:56:27,962 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e260f62d87df4e119066897a7c25673b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680490587058 2023-04-03 02:56:27,971 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#C#compaction#580 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:28,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:28,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:28,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:28,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490648226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:28,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490648230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:28,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490648239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:28,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490648243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:28,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490648248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:28,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490648331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:28,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490648335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,366 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304033a515758d48e43adb84653b333ef6db4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033a515758d48e43adb84653b333ef6db4 2023-04-03 02:56:28,378 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/2171249d0b314b0aa6dbc99b6959d0ec as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2171249d0b314b0aa6dbc99b6959d0ec 2023-04-03 02:56:28,383 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/C of 1af30d70f4e72ebe50e43fde14e4fe9b into 2171249d0b314b0aa6dbc99b6959d0ec(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:28,383 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:28,383 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/C, priority=12, startTime=1680490587911; duration=0sec 2023-04-03 02:56:28,383 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:28,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:28,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490648535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:28,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490648537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,761 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/1b1955a90ec84e81bb37b8adf22337f8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/1b1955a90ec84e81bb37b8adf22337f8 2023-04-03 02:56:28,764 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/A of 1af30d70f4e72ebe50e43fde14e4fe9b into 1b1955a90ec84e81bb37b8adf22337f8(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:28,764 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:28,764 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/A, priority=12, startTime=1680490587911; duration=0sec 2023-04-03 02:56:28,765 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:28,772 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=213, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/27e0e52612a14686a6c913d19fdc65f9 2023-04-03 02:56:28,790 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/081d3e5cd10041e1ab69f7a509e2075d 2023-04-03 02:56:28,810 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/391160c5d50d4e8e9a4c62eb10b96714 2023-04-03 02:56:28,817 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/27e0e52612a14686a6c913d19fdc65f9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/27e0e52612a14686a6c913d19fdc65f9 2023-04-03 02:56:28,820 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/27e0e52612a14686a6c913d19fdc65f9, entries=150, sequenceid=213, filesize=25.3 K 2023-04-03 02:56:28,821 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/081d3e5cd10041e1ab69f7a509e2075d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/081d3e5cd10041e1ab69f7a509e2075d 2023-04-03 02:56:28,824 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/081d3e5cd10041e1ab69f7a509e2075d, entries=150, sequenceid=213, filesize=11.8 K 2023-04-03 02:56:28,825 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/391160c5d50d4e8e9a4c62eb10b96714 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/391160c5d50d4e8e9a4c62eb10b96714 2023-04-03 02:56:28,829 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/391160c5d50d4e8e9a4c62eb10b96714, entries=150, sequenceid=213, filesize=11.8 K 2023-04-03 02:56:28,830 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.09 KB/393312, currentSize=46.96 KB/48090 for 1af30d70f4e72ebe50e43fde14e4fe9b in 906ms, sequenceid=213, compaction requested=false 2023-04-03 02:56:28,830 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:28,830 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:28,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:28,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:28,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:28,830 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:28,835 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,835 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,835 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:28,836 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:28,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:28,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47037f37[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:28,838 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:28,838 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,839 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,839 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:28,839 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:28,839 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:28,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:28,841 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,841 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:28,841 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:28,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 02:56:28,841 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:28,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:28,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:28,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:28,846 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:28,846 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:28,846 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:28,846 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:28,846 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:28,850 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:28,850 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,850 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:28,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:28,852 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,853 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,856 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:28,857 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:28,857 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:28,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:28,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:28,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:28,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:28,857 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:28,857 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:28,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:28,860 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,860 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:28,860 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:28,860 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,860 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,860 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,860 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,860 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304034d584f75555248e19fabfc339d7e8f94 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034d584f75555248e19fabfc339d7e8f94 2023-04-03 02:56:28,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490588859, max timestamp=1680490588878Num uniques 859; 2023-04-03 02:56:28,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:28,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490588859, max timestamp=1680490588878Num uniques 859; 2023-04-03 02:56:28,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:28,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490588859, max timestamp=1680490588878Num uniques 859; 2023-04-03 02:56:28,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:28,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:28,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490648950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:28,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490648951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,957 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:28,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:28,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:28,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:28,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:28,964 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:28,964 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:28,964 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:28,964 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:28,965 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,965 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:28,966 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:28,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:28,966 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:28,967 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,967 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:28,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:28,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:28,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:28,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:28,969 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:28,969 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:28,969 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:28,969 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:28,971 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,971 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:28,972 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,972 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:28,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:28,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785f7fdc[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:28,974 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:28,974 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,975 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:28,975 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,975 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:28,977 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:28,977 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:28,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:28,980 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:28,980 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:28,980 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:28,980 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:28,980 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:28,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:28,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:28,983 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,984 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:28,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,987 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:28,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:28,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:28,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:28,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:28,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:28,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:28,987 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:28,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:28,987 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:28,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:28,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:28,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:28,988 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,988 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:28,988 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490649054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490649054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,088 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:29,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:29,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:29,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:29,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:29,100 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:29,100 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:29,100 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:29,100 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:29,101 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,101 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:29,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:29,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,102 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:29,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:29,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:29,104 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:29,104 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:29,104 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:29,104 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:29,106 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:29,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,107 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:29,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:29,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@326bd5ab[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:29,109 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:29,109 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,110 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,110 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:29,110 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,111 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:29,111 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:29,111 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:29,112 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,112 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:29,112 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:29,112 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:29,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:29,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:29,115 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:29,115 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:29,115 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:29,115 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:29,115 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:29,117 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,117 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:29,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,117 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:29,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:29,118 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,119 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,122 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:29,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:29,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:29,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:29,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:29,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:29,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,123 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:29,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:29,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,123 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:29,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,123 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,223 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:29,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:29,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:29,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:29,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:29,238 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:29,238 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:29,238 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:29,238 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:29,238 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,238 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,239 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:29,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:29,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:29,240 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,240 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:29,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:29,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:29,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:29,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:29,241 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:29,241 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:29,241 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:29,242 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:29,243 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:29,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,243 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739aec52[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:29,245 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:29,245 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490649243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490649249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,255 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:29,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:29,257 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:29,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:29,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:29,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:29,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490649258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,260 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:29,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:29,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:29,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:29,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490649258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:29,260 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:29,262 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,262 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:29,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,262 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:29,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:29,263 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,266 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:29,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:29,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:29,266 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:29,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:29,266 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=228, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/990b1c74667b4921945d3370d50f494c 2023-04-03 02:56:29,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:29,267 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:29,267 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:29,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:29,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:29,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490649262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:29,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:29,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/a25c81f5ec6643989e7521034e303ef9 2023-04-03 02:56:29,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/d031b98e4f414dc988b8e59c189a545f 2023-04-03 02:56:29,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/990b1c74667b4921945d3370d50f494c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/990b1c74667b4921945d3370d50f494c 2023-04-03 02:56:29,328 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/990b1c74667b4921945d3370d50f494c, entries=200, sequenceid=228, filesize=32.1 K 2023-04-03 02:56:29,329 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/a25c81f5ec6643989e7521034e303ef9 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/a25c81f5ec6643989e7521034e303ef9 2023-04-03 02:56:29,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/a25c81f5ec6643989e7521034e303ef9, entries=150, sequenceid=228, filesize=11.8 K 2023-04-03 02:56:29,341 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/d031b98e4f414dc988b8e59c189a545f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/d031b98e4f414dc988b8e59c189a545f 2023-04-03 02:56:29,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/d031b98e4f414dc988b8e59c189a545f, entries=150, sequenceid=228, filesize=11.8 K 2023-04-03 02:56:29,346 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 1af30d70f4e72ebe50e43fde14e4fe9b in 504ms, sequenceid=228, compaction requested=true 2023-04-03 02:56:29,346 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:29,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:29,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:29,346 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:29,346 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:29,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:29,353 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:29,353 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/A is initiating minor compaction (all files) 2023-04-03 02:56:29,353 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/A in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:29,354 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/1b1955a90ec84e81bb37b8adf22337f8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/27e0e52612a14686a6c913d19fdc65f9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/990b1c74667b4921945d3370d50f494c] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=83.1 K 2023-04-03 02:56:29,354 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:29,354 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/B is initiating minor compaction (all files) 2023-04-03 02:56:29,354 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/B in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:29,354 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/58ce3d87840949f481b6df724f8db2f4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/081d3e5cd10041e1ab69f7a509e2075d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/a25c81f5ec6643989e7521034e303ef9] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=35.6 K 2023-04-03 02:56:29,354 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 1b1955a90ec84e81bb37b8adf22337f8, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680490587058 2023-04-03 02:56:29,354 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 58ce3d87840949f481b6df724f8db2f4, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680490587058 2023-04-03 02:56:29,355 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 27e0e52612a14686a6c913d19fdc65f9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680490587121 2023-04-03 02:56:29,355 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 081d3e5cd10041e1ab69f7a509e2075d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680490587121 2023-04-03 02:56:29,355 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 990b1c74667b4921945d3370d50f494c, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680490588225 2023-04-03 02:56:29,355 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting a25c81f5ec6643989e7521034e303ef9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680490588225 2023-04-03 02:56:29,363 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#B#compaction#587 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:29,365 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#A#compaction#586 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:29,367 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:29,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:29,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:29,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:29,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:29,376 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:29,376 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:29,376 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:29,376 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:29,376 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,376 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,379 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:29,379 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:29,379 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,379 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:29,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:29,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:29,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:29,400 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:29,400 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:29,400 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:29,400 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:29,400 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-03 02:56:29,400 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:29,401 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:29,401 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:29,401 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:29,401 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:29,401 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:29,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:29,564 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:29,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490649598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490649598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490649704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490649704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,794 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/98470cd74ff64317911cebf4b5349008 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/98470cd74ff64317911cebf4b5349008 2023-04-03 02:56:29,798 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/6f6228baffbd4bc69659747491d80ece as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6f6228baffbd4bc69659747491d80ece 2023-04-03 02:56:29,798 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/B of 1af30d70f4e72ebe50e43fde14e4fe9b into 98470cd74ff64317911cebf4b5349008(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:29,798 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:29,798 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/B, priority=13, startTime=1680490589346; duration=0sec 2023-04-03 02:56:29,798 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:29,798 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:29,799 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:29,800 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/C is initiating minor compaction (all files) 2023-04-03 02:56:29,800 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/C in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:29,800 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2171249d0b314b0aa6dbc99b6959d0ec, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/391160c5d50d4e8e9a4c62eb10b96714, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/d031b98e4f414dc988b8e59c189a545f] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=35.6 K 2023-04-03 02:56:29,800 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 2171249d0b314b0aa6dbc99b6959d0ec, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680490587058 2023-04-03 02:56:29,800 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 391160c5d50d4e8e9a4c62eb10b96714, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680490587121 2023-04-03 02:56:29,801 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting d031b98e4f414dc988b8e59c189a545f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680490588225 2023-04-03 02:56:29,802 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/A of 1af30d70f4e72ebe50e43fde14e4fe9b into 6f6228baffbd4bc69659747491d80ece(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:29,802 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:29,802 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/A, priority=13, startTime=1680490589346; duration=0sec 2023-04-03 02:56:29,802 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:29,805 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#C#compaction#589 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:29,841 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304033bb63f7cd62b46e1a761d1a990ade5fd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033bb63f7cd62b46e1a761d1a990ade5fd 2023-04-03 02:56:29,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490649907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:29,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:29,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46108 deadline: 1680490649907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:30,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46142 deadline: 1680490650212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:30,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490650212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,215 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/7355f6ad10914002a3f8e5fa3cd65f91 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7355f6ad10914002a3f8e5fa3cd65f91 2023-04-03 02:56:30,218 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/C of 1af30d70f4e72ebe50e43fde14e4fe9b into 7355f6ad10914002a3f8e5fa3cd65f91(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:30,219 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:30,219 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/C, priority=13, startTime=1680490589346; duration=0sec 2023-04-03 02:56:30,219 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:30,254 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=262, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/8d2aa5abed4b4e6b8e318bb2643adf6a 2023-04-03 02:56:30,664 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/97c9b7ea81f5422eb767be7d5b869556 2023-04-03 02:56:30,689 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/143a8567b9fa4363a52f8843e34704c0 2023-04-03 02:56:30,694 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/8d2aa5abed4b4e6b8e318bb2643adf6a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/8d2aa5abed4b4e6b8e318bb2643adf6a 2023-04-03 02:56:30,698 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/8d2aa5abed4b4e6b8e318bb2643adf6a, entries=150, sequenceid=262, filesize=25.4 K 2023-04-03 02:56:30,700 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/97c9b7ea81f5422eb767be7d5b869556 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/97c9b7ea81f5422eb767be7d5b869556 2023-04-03 02:56:30,709 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/97c9b7ea81f5422eb767be7d5b869556, entries=150, sequenceid=262, filesize=11.9 K 2023-04-03 02:56:30,710 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/143a8567b9fa4363a52f8843e34704c0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/143a8567b9fa4363a52f8843e34704c0 2023-04-03 02:56:30,713 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/143a8567b9fa4363a52f8843e34704c0, entries=150, sequenceid=262, filesize=11.9 K 2023-04-03 02:56:30,714 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 1af30d70f4e72ebe50e43fde14e4fe9b in 1314ms, sequenceid=262, compaction requested=false 2023-04-03 02:56:30,714 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:30,714 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:30,716 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,716 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:30,716 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,716 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbcdfd1[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:30,719 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:30,719 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:30,720 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 02:56:30,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:30,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:30,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:30,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:30,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:30,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:30,721 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,721 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:30,721 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:30,721 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:30,721 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,721 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:30,723 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:30,723 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:30,723 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:30,723 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:30,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:30,727 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:30,727 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:30,727 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:30,727 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:30,727 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:30,730 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:30,730 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,730 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:30,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,731 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,733 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:30,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:30,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,736 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:30,736 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:30,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:30,736 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403043e767c37ce4476a0e7986c2cc3b15f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403043e767c37ce4476a0e7986c2cc3b15f 2023-04-03 02:56:30,736 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:30,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:30,736 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:30,737 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:30,737 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:30,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:30,736 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:30,742 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:30,742 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,743 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,743 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,743 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:30,744 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=279, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/fb4b1cef68844bc2b58c3d6638178f0a 2023-04-03 02:56:30,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:30,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490650747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:30,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490650748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,837 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:30,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:30,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:30,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:30,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:30,840 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:30,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:30,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:30,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:30,841 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,841 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:30,842 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:30,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:30,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:30,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,842 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:30,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:30,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:30,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:30,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:30,844 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:30,844 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:30,844 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:30,848 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:30,848 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:30,848 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:30,848 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:30,848 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:30,848 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:30,848 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:30,848 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:30,848 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,848 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:30,848 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:30,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:30,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490650851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:30,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490650851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,859 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,859 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:30,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:30,859 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,859 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:30,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:30,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c1ef465[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:30,862 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:30,862 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,865 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,865 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:30,865 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:30,865 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:30,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:30,867 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:30,867 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:30,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:30,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:30,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:30,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:30,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:30,870 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:30,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:30,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:30,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:30,870 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:30,871 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,871 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,871 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:30,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:30,872 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:30,873 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:30,875 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,875 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:30,875 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:30,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:30,875 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:30,875 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:30,875 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:30,875 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,875 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,875 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,875 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:30,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:30,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:30,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:30,876 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:30,876 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:30,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:30,976 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:30,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:30,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:30,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:30,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:30,980 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:30,980 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:30,980 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:30,980 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:30,981 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,981 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:30,983 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:30,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:30,983 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:30,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:30,984 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,984 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:30,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:30,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:30,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:30,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:30,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:30,989 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:30,989 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:30,989 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:30,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:30,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:30,996 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:30,996 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:30,996 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:30,996 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:30,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:30,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:30,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:30,996 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:30,998 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:30,999 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:31,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40b4aac0[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:31,001 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:31,001 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,002 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,002 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:31,002 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:31,002 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:31,002 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,002 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:31,004 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,004 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:31,004 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,004 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:31,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:31,007 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:31,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:31,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:31,007 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:31,007 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:31,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:31,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,018 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,020 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:31,023 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:31,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:31,023 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:31,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:31,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:31,024 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:31,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:31,024 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:31,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:31,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490651054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:31,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490651055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,124 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:31,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:31,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:31,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:31,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:31,127 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:31,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:31,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:31,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:31,128 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,128 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:31,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:31,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:31,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:31,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:31,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:31,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:31,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:31,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:31,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:31,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:31,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:31,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:31,130 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,130 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:31,130 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:31,130 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:31,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:31,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:31,131 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:31,132 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,132 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e836f8[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:31,133 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:31,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:31,136 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,136 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:31,137 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:31,137 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:31,137 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,137 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:31,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:31,139 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:31,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:31,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:31,139 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:31,139 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:31,140 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,140 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,140 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:31,141 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,142 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:31,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:31,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:31,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:31,144 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:31,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:31,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:31,145 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,145 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,145 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:31,145 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:31,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/64f6537b11ba479fb2b1ae907fa367a3 2023-04-03 02:56:31,245 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:31,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:31,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:31,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:31,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:31,249 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:31,249 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:31,249 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:31,249 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:31,249 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,250 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,252 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,252 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:31,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:31,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:31,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:31,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:31,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:31,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:31,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490651254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:31,255 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:31,255 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:31,255 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:31,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:31,257 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:31,257 DEBUG [Thread-3908] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4129 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:56:31,257 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,257 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:31,258 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:31,258 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:31,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,259 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,260 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490651257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,262 DEBUG [Thread-3904] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:56:31,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:31,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de32fe3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:31,262 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:31,262 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,264 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:31,264 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:31,264 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:31,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,264 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:31,265 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,265 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:31,265 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:31,266 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:31,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:31,269 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:31,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:31,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:31,269 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:31,269 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:31,272 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,272 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,272 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:31,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,276 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,286 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,286 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,287 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:31,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:31,286 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:31,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:31,287 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,287 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:31,287 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:31,287 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:31,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:31,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:31,287 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,287 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,287 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:31,287 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:31,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:31,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:31,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490651287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,291 DEBUG [Thread-3912] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:56:31,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:31,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490651357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:31,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490651359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,387 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:31,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:31,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:31,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:31,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:31,390 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:31,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:31,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:31,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:31,391 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,391 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,392 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:31,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:31,392 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,392 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,392 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:31,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:31,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:31,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:31,394 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,394 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:31,394 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:31,394 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:31,395 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,395 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,395 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,395 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76054be6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:31,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:31,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,399 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:31,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:31,399 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,399 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:31,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:31,400 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:31,401 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,401 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:31,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:31,403 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:31,403 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:31,403 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:31,403 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:31,403 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:31,404 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,404 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,404 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:31,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,405 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,406 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:31,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:31,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:31,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:31,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:31,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:31,412 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:31,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:31,412 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:31,412 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,413 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:31,513 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:31,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:31,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:31,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:31,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:31,516 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:31,516 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:31,516 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:31,516 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:31,516 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,516 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,517 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,517 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:31,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:31,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:31,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:31,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:31,519 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:31,519 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:31,519 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:31,519 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:31,519 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:31,520 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,520 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:31,520 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:31,520 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:31,521 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,522 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,522 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13820d4c[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:31,523 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:31,523 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,524 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:31,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:31,524 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,524 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:31,524 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:31,531 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,532 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:31,534 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:31,534 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:31,534 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:31,534 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:31,534 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:31,535 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:31,535 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,535 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:31,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:31,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:31,536 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,537 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:31,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:31,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:31,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:31,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:31,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:31,539 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:31,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:31,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:31,539 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,539 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:31,539 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:31,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:31,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/c1b41041271e439dbe3caed62b43e5bd 2023-04-03 02:56:31,565 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/fb4b1cef68844bc2b58c3d6638178f0a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/fb4b1cef68844bc2b58c3d6638178f0a 2023-04-03 02:56:31,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/fb4b1cef68844bc2b58c3d6638178f0a, entries=200, sequenceid=279, filesize=32.3 K 2023-04-03 02:56:31,569 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/64f6537b11ba479fb2b1ae907fa367a3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/64f6537b11ba479fb2b1ae907fa367a3 2023-04-03 02:56:31,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/64f6537b11ba479fb2b1ae907fa367a3, entries=150, sequenceid=279, filesize=11.9 K 2023-04-03 02:56:31,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/c1b41041271e439dbe3caed62b43e5bd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c1b41041271e439dbe3caed62b43e5bd 2023-04-03 02:56:31,576 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c1b41041271e439dbe3caed62b43e5bd, entries=150, sequenceid=279, filesize=11.9 K 2023-04-03 02:56:31,576 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1af30d70f4e72ebe50e43fde14e4fe9b in 856ms, sequenceid=279, compaction requested=true 2023-04-03 02:56:31,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:31,577 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:31,577 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:31,577 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:31,577 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:31,577 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:31,577 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:31,578 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85506 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:31,578 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/B is initiating minor compaction (all files) 2023-04-03 02:56:31,578 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/A is initiating minor compaction (all files) 2023-04-03 02:56:31,578 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/B in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,578 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/A in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,578 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/98470cd74ff64317911cebf4b5349008, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/97c9b7ea81f5422eb767be7d5b869556, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/64f6537b11ba479fb2b1ae907fa367a3] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=36.0 K 2023-04-03 02:56:31,578 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6f6228baffbd4bc69659747491d80ece, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/8d2aa5abed4b4e6b8e318bb2643adf6a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/fb4b1cef68844bc2b58c3d6638178f0a] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=83.5 K 2023-04-03 02:56:31,578 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 98470cd74ff64317911cebf4b5349008, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680490588225 2023-04-03 02:56:31,578 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6f6228baffbd4bc69659747491d80ece, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680490588225 2023-04-03 02:56:31,578 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 97c9b7ea81f5422eb767be7d5b869556, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1680490588938 2023-04-03 02:56:31,579 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 8d2aa5abed4b4e6b8e318bb2643adf6a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1680490588938 2023-04-03 02:56:31,579 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 64f6537b11ba479fb2b1ae907fa367a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680490589591 2023-04-03 02:56:31,579 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting fb4b1cef68844bc2b58c3d6638178f0a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680490589591 2023-04-03 02:56:31,585 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#B#compaction#596 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:31,589 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#A#compaction#595 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:31,615 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/6681df7ddeba44b49807af96378cbbee as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/6681df7ddeba44b49807af96378cbbee 2023-04-03 02:56:31,619 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/B of 1af30d70f4e72ebe50e43fde14e4fe9b into 6681df7ddeba44b49807af96378cbbee(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:31,619 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:31,619 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/B, priority=13, startTime=1680490591577; duration=0sec 2023-04-03 02:56:31,619 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:31,619 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:31,620 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:31,620 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/C is initiating minor compaction (all files) 2023-04-03 02:56:31,620 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/C in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,620 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7355f6ad10914002a3f8e5fa3cd65f91, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/143a8567b9fa4363a52f8843e34704c0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c1b41041271e439dbe3caed62b43e5bd] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=36.0 K 2023-04-03 02:56:31,620 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7355f6ad10914002a3f8e5fa3cd65f91, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680490588225 2023-04-03 02:56:31,621 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 143a8567b9fa4363a52f8843e34704c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1680490588938 2023-04-03 02:56:31,621 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting c1b41041271e439dbe3caed62b43e5bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680490589591 2023-04-03 02:56:31,626 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#C#compaction#597 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:31,639 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:31,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:31,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:31,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:31,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:31,643 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:31,643 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:31,643 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:31,643 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:31,643 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,643 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,644 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:31,644 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:31,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:31,645 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,645 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:31,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:31,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:31,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:31,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:31,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:31,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:31,646 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:31,646 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:31,646 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:31,646 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:31,646 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:31,646 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:31,646 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:31,647 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 02:56:31,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:31,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:31,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:31,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:31,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:31,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:31,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:31,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:31,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:31,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490651881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:31,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490651881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:31,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490651984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:31,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:31,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490651984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,045 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/98028ab3ab524315a965f7f91d2b0061 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/98028ab3ab524315a965f7f91d2b0061 2023-04-03 02:56:32,049 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/C of 1af30d70f4e72ebe50e43fde14e4fe9b into 98028ab3ab524315a965f7f91d2b0061(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:32,049 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:32,049 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/C, priority=13, startTime=1680490591577; duration=0sec 2023-04-03 02:56:32,050 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:32,055 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/44df0498b8344ec7965f77465f780e04 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/44df0498b8344ec7965f77465f780e04 2023-04-03 02:56:32,059 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/A of 1af30d70f4e72ebe50e43fde14e4fe9b into 44df0498b8344ec7965f77465f780e04(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:32,059 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:32,059 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/A, priority=13, startTime=1680490591576; duration=0sec 2023-04-03 02:56:32,059 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:32,073 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040336b10d80ae504ab2b9b4bd60021f2b1e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040336b10d80ae504ab2b9b4bd60021f2b1e 2023-04-03 02:56:32,079 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=298, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/85af34c3d8454c1fafe755fdaf263fde 2023-04-03 02:56:32,089 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/0c9cbb3285f74632b733fcf9311bf0c0 2023-04-03 02:56:32,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:32,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490652186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:32,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490652189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:32,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490652490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:32,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490652491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,497 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/c53b7b99bf1b4c2f9b0f89552243e9cb 2023-04-03 02:56:32,500 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/85af34c3d8454c1fafe755fdaf263fde as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/85af34c3d8454c1fafe755fdaf263fde 2023-04-03 02:56:32,503 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/85af34c3d8454c1fafe755fdaf263fde, entries=150, sequenceid=298, filesize=25.4 K 2023-04-03 02:56:32,504 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/0c9cbb3285f74632b733fcf9311bf0c0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/0c9cbb3285f74632b733fcf9311bf0c0 2023-04-03 02:56:32,509 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/0c9cbb3285f74632b733fcf9311bf0c0, entries=150, sequenceid=298, filesize=11.9 K 2023-04-03 02:56:32,509 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/c53b7b99bf1b4c2f9b0f89552243e9cb as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c53b7b99bf1b4c2f9b0f89552243e9cb 2023-04-03 02:56:32,513 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c53b7b99bf1b4c2f9b0f89552243e9cb, entries=150, sequenceid=298, filesize=11.9 K 2023-04-03 02:56:32,513 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 1af30d70f4e72ebe50e43fde14e4fe9b in 867ms, sequenceid=298, compaction requested=false 2023-04-03 02:56:32,514 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:32,514 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:32,514 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:32,514 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:32,514 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,514 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:32,514 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:32,517 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:32,517 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:32,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:32,517 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:32,517 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:32,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:32,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f8a632[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:32,525 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:32,525 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:32,526 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:32,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:32,526 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:32,526 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:32,527 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:32,527 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,527 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:32,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:32,528 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:32,528 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:32,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:32,531 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:32,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:32,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:32,531 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:32,531 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:32,533 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:32,533 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:32,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:32,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:32,534 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:32,535 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:32,540 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:32,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:32,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,554 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:32,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:32,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:32,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:32,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:32,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:32,554 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:32,554 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:32,554 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:32,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:32,557 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:32,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:32,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:32,654 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:32,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:32,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:32,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:32,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:32,658 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:32,658 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:32,658 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:32,658 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:32,658 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:32,658 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:32,659 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,659 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:32,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:32,660 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:32,660 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:32,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:32,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:32,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:32,661 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:32,661 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:32,661 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:32,661 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:32,661 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:32,662 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:32,662 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:32,662 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 02:56:32,662 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:32,662 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:32,662 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:32,662 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:32,662 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:32,662 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:32,677 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a0a8fa6a80534d988c0184e92ee77651 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a0a8fa6a80534d988c0184e92ee77651 2023-04-03 02:56:32,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:32,997 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:33,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:33,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490653019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:33,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490653020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,093 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=315, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/81e0cd4510b14286a69ba12a86f99cf6 2023-04-03 02:56:33,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:33,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490653122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:33,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490653122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:33,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490653324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:33,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490653325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,512 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/370dd50ba7e544748e4db8d0baa85365 2023-04-03 02:56:33,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:33,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490653627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:33,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490653630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,926 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/837836bd13f74c619bdb2201e9abb941 2023-04-03 02:56:33,930 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/81e0cd4510b14286a69ba12a86f99cf6 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/81e0cd4510b14286a69ba12a86f99cf6 2023-04-03 02:56:33,933 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/81e0cd4510b14286a69ba12a86f99cf6, entries=150, sequenceid=315, filesize=25.4 K 2023-04-03 02:56:33,933 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/370dd50ba7e544748e4db8d0baa85365 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/370dd50ba7e544748e4db8d0baa85365 2023-04-03 02:56:33,936 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/370dd50ba7e544748e4db8d0baa85365, entries=150, sequenceid=315, filesize=11.9 K 2023-04-03 02:56:33,937 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/837836bd13f74c619bdb2201e9abb941 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/837836bd13f74c619bdb2201e9abb941 2023-04-03 02:56:33,940 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/837836bd13f74c619bdb2201e9abb941, entries=150, sequenceid=315, filesize=11.9 K 2023-04-03 02:56:33,940 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 1af30d70f4e72ebe50e43fde14e4fe9b in 1278ms, sequenceid=315, compaction requested=true 2023-04-03 02:56:33,940 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:33,940 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:33,940 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:33,942 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:33,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:33,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:33,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:33,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:33,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2500b39[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:33,944 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:33,944 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:33,945 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:33,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:33,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:33,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:33,945 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:33,945 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,945 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:33,947 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,947 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:33,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:33,947 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:33,950 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:33,950 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:33,950 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:33,950 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:33,950 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:33,951 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:33,951 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:33,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:33,951 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:33,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:33,952 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:33,953 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:33,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:33,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:33,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:33,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:33,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:33,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:33,957 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:33,957 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:33,957 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:33,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:33,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:33,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:33,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:33,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:33,957 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,057 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:34,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:34,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:34,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:34,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:34,060 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:34,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:34,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:34,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,061 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,062 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,062 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,062 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,063 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:34,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:34,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:34,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:34,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:34,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:34,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:34,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:34,064 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:34,064 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,064 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:34,064 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 02:56:34,064 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:34,064 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:34,064 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:34,064 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:34,064 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:34,064 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:34,075 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040313317be73adf4cee85fdbaadbfe26d3d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040313317be73adf4cee85fdbaadbfe26d3d 2023-04-03 02:56:34,089 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=332, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/278d59f86c9447d2a6ae501235e7259f 2023-04-03 02:56:34,102 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/85ebc1ace59149318d6647ceedbafe97 2023-04-03 02:56:34,111 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/b5f2987e89da4cd8bd80ab7da5a452bb 2023-04-03 02:56:34,115 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/278d59f86c9447d2a6ae501235e7259f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/278d59f86c9447d2a6ae501235e7259f 2023-04-03 02:56:34,118 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/278d59f86c9447d2a6ae501235e7259f, entries=150, sequenceid=332, filesize=25.4 K 2023-04-03 02:56:34,118 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/85ebc1ace59149318d6647ceedbafe97 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/85ebc1ace59149318d6647ceedbafe97 2023-04-03 02:56:34,122 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/85ebc1ace59149318d6647ceedbafe97, entries=150, sequenceid=332, filesize=11.9 K 2023-04-03 02:56:34,122 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/b5f2987e89da4cd8bd80ab7da5a452bb as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b5f2987e89da4cd8bd80ab7da5a452bb 2023-04-03 02:56:34,127 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b5f2987e89da4cd8bd80ab7da5a452bb, entries=150, sequenceid=332, filesize=11.9 K 2023-04-03 02:56:34,127 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 1af30d70f4e72ebe50e43fde14e4fe9b in 63ms, sequenceid=332, compaction requested=true 2023-04-03 02:56:34,127 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:34,128 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:34,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,129 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,130 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:34,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e562aee[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:34,131 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:34,132 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,134 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:34,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:34,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:34,134 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:34,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,135 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,136 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:34,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:34,138 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:34,138 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:34,138 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:34,138 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:34,138 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:34,139 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,139 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,140 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,140 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,141 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:34,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:34,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,144 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:34,143 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:34,144 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:34,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,144 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:34,165 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:56:34,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:34,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:34,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:34,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:34,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:34,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:34,180 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403e10d32a58c114019a31e51300202d9fe to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e10d32a58c114019a31e51300202d9fe 2023-04-03 02:56:34,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490594197, max timestamp=1680490594218Num uniques 859; 2023-04-03 02:56:34,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490594197, max timestamp=1680490594218Num uniques 859; 2023-04-03 02:56:34,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490594197, max timestamp=1680490594218Num uniques 859; 2023-04-03 02:56:34,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:34,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:34,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:34,244 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:34,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:34,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:34,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:34,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:34,247 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:34,247 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,247 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:34,247 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:34,248 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,248 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,250 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,250 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,250 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,250 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:34,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:34,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:34,251 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:34,251 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,251 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:34,251 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:34,251 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:34,251 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,251 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:34,251 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:34,251 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,252 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:34,252 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:34,252 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,252 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:34,252 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:34,253 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,253 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,253 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,253 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:34,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbda405[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:34,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:34,255 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,256 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:34,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:34,256 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:34,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:34,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:34,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490654253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:34,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490654254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,257 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,258 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:34,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:34,260 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:34,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:34,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:34,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:34,260 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:34,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,263 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,263 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,265 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,267 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:34,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,267 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:34,267 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,268 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:34,268 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,268 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:34,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490654358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:34,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490654358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,368 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:34,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:34,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:34,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:34,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:34,370 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:34,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:34,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:34,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,372 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,372 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,372 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,372 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:34,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:34,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:34,373 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:34,373 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,373 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:34,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:34,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:34,374 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,374 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:34,374 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:34,374 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:34,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:34,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:34,374 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:34,375 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,375 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,376 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,376 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:34,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e8750b6[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:34,377 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:34,377 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,384 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:34,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:34,384 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:34,385 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:34,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:34,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,386 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:34,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:34,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:34,388 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:34,388 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:34,388 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:34,388 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:34,388 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:34,390 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,390 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,390 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,390 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,391 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:34,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:34,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,393 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,394 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,394 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,394 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:34,393 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,394 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:34,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:34,394 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,494 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:34,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:34,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:34,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:34,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:34,498 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:34,498 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,498 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:34,498 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:34,498 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,498 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,501 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,501 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,502 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:34,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:34,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:34,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:34,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:34,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,503 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:34,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:34,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:34,504 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,504 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:34,504 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:34,504 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:34,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:34,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:34,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:34,508 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,508 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,509 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,509 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:34,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:34,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c0c8f9c[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:34,510 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:34,510 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,511 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,511 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:34,511 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:34,511 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:34,511 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:34,513 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:34,513 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,513 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,513 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:34,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:34,515 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:34,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:34,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:34,515 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:34,515 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:34,517 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,519 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,519 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,519 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,525 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,525 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,525 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,525 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:34,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:34,525 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,525 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,526 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,526 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:34,526 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:34,526 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,526 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,526 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:34,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490654561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:34,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490654562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,586 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=343, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/c0491d4fe9004645acf138a25d59902c 2023-04-03 02:56:34,595 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/ca0cc23cef8541d3ade2ccb72dd730dc 2023-04-03 02:56:34,603 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/c8635e38ed3249ae8ec187dace9f8da3 2023-04-03 02:56:34,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/c0491d4fe9004645acf138a25d59902c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/c0491d4fe9004645acf138a25d59902c 2023-04-03 02:56:34,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/c0491d4fe9004645acf138a25d59902c, entries=200, sequenceid=343, filesize=32.3 K 2023-04-03 02:56:34,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/ca0cc23cef8541d3ade2ccb72dd730dc as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ca0cc23cef8541d3ade2ccb72dd730dc 2023-04-03 02:56:34,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ca0cc23cef8541d3ade2ccb72dd730dc, entries=150, sequenceid=343, filesize=11.9 K 2023-04-03 02:56:34,614 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/c8635e38ed3249ae8ec187dace9f8da3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c8635e38ed3249ae8ec187dace9f8da3 2023-04-03 02:56:34,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c8635e38ed3249ae8ec187dace9f8da3, entries=150, sequenceid=343, filesize=11.9 K 2023-04-03 02:56:34,617 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1af30d70f4e72ebe50e43fde14e4fe9b in 452ms, sequenceid=343, compaction requested=true 2023-04-03 02:56:34,617 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:34,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:34,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:34,617 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:56:34,617 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:56:34,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:34,618 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:56:34,618 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/A is initiating minor compaction (all files) 2023-04-03 02:56:34,618 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61510 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:56:34,618 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/A in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,618 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/B is initiating minor compaction (all files) 2023-04-03 02:56:34,619 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/44df0498b8344ec7965f77465f780e04, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/85af34c3d8454c1fafe755fdaf263fde, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/81e0cd4510b14286a69ba12a86f99cf6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/278d59f86c9447d2a6ae501235e7259f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/c0491d4fe9004645acf138a25d59902c] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=134.6 K 2023-04-03 02:56:34,619 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/B in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,619 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/6681df7ddeba44b49807af96378cbbee, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/0c9cbb3285f74632b733fcf9311bf0c0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/370dd50ba7e544748e4db8d0baa85365, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/85ebc1ace59149318d6647ceedbafe97, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ca0cc23cef8541d3ade2ccb72dd730dc] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=60.1 K 2023-04-03 02:56:34,619 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 44df0498b8344ec7965f77465f780e04, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680490589591 2023-04-03 02:56:34,619 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 6681df7ddeba44b49807af96378cbbee, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680490589591 2023-04-03 02:56:34,619 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 85af34c3d8454c1fafe755fdaf263fde, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680490590738 2023-04-03 02:56:34,619 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 0c9cbb3285f74632b733fcf9311bf0c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680490590738 2023-04-03 02:56:34,619 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 81e0cd4510b14286a69ba12a86f99cf6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680490591874 2023-04-03 02:56:34,620 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 370dd50ba7e544748e4db8d0baa85365, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680490591874 2023-04-03 02:56:34,620 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 278d59f86c9447d2a6ae501235e7259f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680490593008 2023-04-03 02:56:34,620 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 85ebc1ace59149318d6647ceedbafe97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680490593008 2023-04-03 02:56:34,620 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting c0491d4fe9004645acf138a25d59902c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680490594145 2023-04-03 02:56:34,620 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ca0cc23cef8541d3ade2ccb72dd730dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680490594146 2023-04-03 02:56:34,626 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:34,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:34,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:34,629 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#B#compaction#611 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:34,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:34,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:34,630 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:34,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:34,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:34,632 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#A#compaction#610 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:34,632 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,632 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,633 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,633 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,634 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,634 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:34,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:34,637 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/57e4a60eb1fd4834b5656e359a2eda8d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/57e4a60eb1fd4834b5656e359a2eda8d 2023-04-03 02:56:34,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:34,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:34,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:34,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:34,645 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/B of 1af30d70f4e72ebe50e43fde14e4fe9b into 57e4a60eb1fd4834b5656e359a2eda8d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:34,645 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:34,645 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/B, priority=11, startTime=1680490594617; duration=0sec 2023-04-03 02:56:34,645 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:34,645 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 02:56:34,646 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61510 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 02:56:34,646 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/C is initiating minor compaction (all files) 2023-04-03 02:56:34,646 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/C in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,646 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/98028ab3ab524315a965f7f91d2b0061, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c53b7b99bf1b4c2f9b0f89552243e9cb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/837836bd13f74c619bdb2201e9abb941, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b5f2987e89da4cd8bd80ab7da5a452bb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c8635e38ed3249ae8ec187dace9f8da3] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=60.1 K 2023-04-03 02:56:34,646 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 98028ab3ab524315a965f7f91d2b0061, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1680490589591 2023-04-03 02:56:34,647 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting c53b7b99bf1b4c2f9b0f89552243e9cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680490590738 2023-04-03 02:56:34,647 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 837836bd13f74c619bdb2201e9abb941, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1680490591874 2023-04-03 02:56:34,647 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:34,647 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:34,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:34,647 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b5f2987e89da4cd8bd80ab7da5a452bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680490593008 2023-04-03 02:56:34,647 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-03 02:56:34,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:34,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:34,647 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting c8635e38ed3249ae8ec187dace9f8da3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680490594146 2023-04-03 02:56:34,647 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:34,648 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:34,648 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:34,648 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:34,658 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#C#compaction#613 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:34,662 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403329145d337934fee974795953efd3cb7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403329145d337934fee974795953efd3cb7 2023-04-03 02:56:34,667 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=379, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/67a36d089e9e441c8c897c5ae67ef092 2023-04-03 02:56:34,669 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/ef50a5f70fff4b7e8b0f84fe61cb4d0e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/ef50a5f70fff4b7e8b0f84fe61cb4d0e 2023-04-03 02:56:34,672 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/C of 1af30d70f4e72ebe50e43fde14e4fe9b into ef50a5f70fff4b7e8b0f84fe61cb4d0e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:34,672 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:34,672 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/C, priority=11, startTime=1680490594617; duration=0sec 2023-04-03 02:56:34,673 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:34,675 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/220d437b71fe4acd9c0c43f89cd328e3 2023-04-03 02:56:34,682 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/7fe5b47aafa244bf83a2abd1d51dd1aa 2023-04-03 02:56:34,684 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/67a36d089e9e441c8c897c5ae67ef092 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/67a36d089e9e441c8c897c5ae67ef092 2023-04-03 02:56:34,687 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/67a36d089e9e441c8c897c5ae67ef092, entries=150, sequenceid=379, filesize=25.4 K 2023-04-03 02:56:34,687 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/220d437b71fe4acd9c0c43f89cd328e3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/220d437b71fe4acd9c0c43f89cd328e3 2023-04-03 02:56:34,690 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/220d437b71fe4acd9c0c43f89cd328e3, entries=150, sequenceid=379, filesize=11.9 K 2023-04-03 02:56:34,691 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/7fe5b47aafa244bf83a2abd1d51dd1aa as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7fe5b47aafa244bf83a2abd1d51dd1aa 2023-04-03 02:56:34,694 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7fe5b47aafa244bf83a2abd1d51dd1aa, entries=150, sequenceid=379, filesize=11.9 K 2023-04-03 02:56:34,695 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 1af30d70f4e72ebe50e43fde14e4fe9b in 48ms, sequenceid=379, compaction requested=false 2023-04-03 02:56:34,695 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:34,695 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:34,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:34,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:34,695 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:34,696 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,696 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,697 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:34,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43e2d1a9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:34,709 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:34,709 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,711 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,711 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:34,711 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,711 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:34,711 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:34,712 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:34,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:34,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,713 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,713 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:34,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:34,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:34,717 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:34,717 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:34,717 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:34,717 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:34,717 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:34,719 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,719 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,719 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,720 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,731 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,733 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,733 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,733 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,733 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,733 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,733 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:34,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:34,733 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,734 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,734 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,734 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:34,733 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,734 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,834 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:34,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:34,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:34,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:34,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:34,837 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:34,837 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,837 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:34,837 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:34,838 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,838 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,840 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,840 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:34,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:34,841 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:34,841 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,841 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:34,842 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,842 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:34,842 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:34,842 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,843 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,844 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,844 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73cfd85c[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:34,845 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:34,846 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,846 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,847 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:34,847 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:34,847 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:34,847 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,847 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:34,848 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,848 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:34,848 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,848 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:34,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:34,850 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:34,850 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:34,850 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:34,851 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:34,851 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:34,854 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,854 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,854 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,855 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,856 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,858 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,858 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,858 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,858 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,858 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,858 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,858 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:34,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:34,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:34,858 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,859 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,859 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:34,859 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,859 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:34,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:34,902 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 02:56:34,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:34,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:34,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:34,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:34,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:34,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:34,915 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403217b47190142476bb064300e09b37f75 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403217b47190142476bb064300e09b37f75 2023-04-03 02:56:34,920 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=392, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/373414e4ed3a421cb527571bd0d6ca22 2023-04-03 02:56:34,959 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:34,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:34,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:34,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:34,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:34,962 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:34,962 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,962 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:34,962 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:34,962 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,963 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,963 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,963 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,964 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,964 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:34,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:34,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:34,965 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:34,965 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:34,965 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:34,967 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,967 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:34,967 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:34,967 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:34,969 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:34,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67593473[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:34,971 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:34,971 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,976 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:34,977 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,977 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:34,978 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:34,978 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,978 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,978 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:34,980 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:34,980 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:34,981 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:34,981 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:34,981 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:34,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,982 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:34,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,983 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,984 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,988 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:34,988 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:34,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:34,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:34,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:34,988 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:34,988 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:34,988 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:34,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:34,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490594941, max timestamp=1680490594988Num uniques 859; 2023-04-03 02:56:34,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:56:34,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680490594941, max timestamp=1680490594988Num uniques 859; 2023-04-03 02:56:34,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680490594941, max timestamp=1680490594988Num uniques 859; 2023-04-03 02:56:34,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:34,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 02:56:34,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 02:56:34,990 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,990 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,990 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:34,990 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:34,990 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,990 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:34,990 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:35,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490655016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:35,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490655019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,060 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/6a76f7c9453347ec924657b7517e2f6f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6a76f7c9453347ec924657b7517e2f6f 2023-04-03 02:56:35,064 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/A of 1af30d70f4e72ebe50e43fde14e4fe9b into 6a76f7c9453347ec924657b7517e2f6f(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:35,064 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:35,064 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/A, priority=11, startTime=1680490594617; duration=0sec 2023-04-03 02:56:35,064 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:35,088 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:35,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:35,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:35,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:35,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:35,092 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:35,092 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,092 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:35,092 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:35,092 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,092 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,093 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,093 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,094 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,094 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:35,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:35,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:35,095 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:35,095 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,095 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:35,095 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:35,095 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:35,095 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,095 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:35,095 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:35,095 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,096 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:35,096 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:35,096 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,096 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:35,096 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:35,097 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,097 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,097 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,097 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:35,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f61376b[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:35,099 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:35,099 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,100 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:35,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:35,100 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:35,100 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,100 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:35,102 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,102 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:35,102 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,102 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:35,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:35,105 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:35,105 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:35,105 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:35,105 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:35,105 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:35,106 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,106 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,106 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,107 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,108 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,110 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,110 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,110 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,110 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,111 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,111 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:35,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:35,111 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,111 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,111 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:35,111 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:35,111 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:35,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:35,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490655120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:35,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490655122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,211 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:35,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:35,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:35,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:35,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:35,218 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:35,218 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,218 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:35,218 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:35,218 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,218 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,219 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:35,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:35,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:35,221 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:35,221 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,221 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:35,221 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:35,221 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:35,221 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,221 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:35,221 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:35,221 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,221 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,221 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:35,222 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:35,222 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,222 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:35,222 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:35,222 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,222 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:35,224 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,224 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,224 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,224 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:35,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7adbd4b7[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:35,225 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:35,225 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,226 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:35,227 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,227 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:35,228 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,228 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:35,228 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,228 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:35,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:35,230 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:35,230 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:35,230 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:35,231 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:35,231 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:35,232 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,232 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,232 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,234 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,238 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,238 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,239 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:35,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:35,239 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:35,239 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:35,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:35,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:35,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46104 deadline: 1680490655271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,273 DEBUG [Thread-3904] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:56:35,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:35,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46064 deadline: 1680490655290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,293 DEBUG [Thread-3908] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:56:35,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:35,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46080 deadline: 1680490655311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,315 DEBUG [Thread-3912] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., hostname=jenkins-hbase4.apache.org,43815,1680490421348, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 02:56:35,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:35,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490655324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:35,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490655324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,340 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:35,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:35,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:35,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:35,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:35,343 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:35,343 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,343 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:35,343 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:35,343 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,343 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,352 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,352 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,352 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,352 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/832782a2a1634a109eca1c6a171101cd 2023-04-03 02:56:35,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:35,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:35,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:35,354 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,354 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:35,354 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:35,354 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:35,356 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,357 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,357 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77dbee88[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:35,359 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:35,359 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,360 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:35,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:35,360 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,360 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:35,360 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:35,362 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,362 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,362 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,363 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:35,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:35,365 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:35,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:35,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:35,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:35,365 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:35,369 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,369 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,369 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,369 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,371 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,373 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,373 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:35,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:35,373 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:35,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,373 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,373 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,373 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:35,373 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,374 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,374 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,374 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:35,374 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,374 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,474 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:35,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:35,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:35,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:35,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:35,483 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:35,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:35,483 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:35,484 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,484 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,485 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,485 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,485 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,485 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:35,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:35,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:35,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:35,486 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:35,486 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,486 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:35,486 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:35,486 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:35,487 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,487 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:35,487 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:35,487 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,487 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:35,487 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:35,487 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,487 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:35,487 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:35,490 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,490 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,490 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,490 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:35,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@590fb6ac[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:35,492 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:35,492 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,493 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,493 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:35,493 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:35,493 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:35,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:35,494 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:35,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:35,497 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:35,497 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:35,497 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:35,497 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:35,497 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:35,498 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,498 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,498 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,499 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,500 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,503 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,503 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,503 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,503 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,503 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,504 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,503 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:35,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:35,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,504 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:35,504 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:35,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,604 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:35,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:35,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:35,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:35,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:35,607 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:35,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:35,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:35,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,608 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,609 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,609 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,609 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,609 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:35,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:35,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:35,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:35,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:35,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:35,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:35,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:35,611 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,611 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:35,611 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:35,611 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:35,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:35,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:35,611 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:35,612 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,612 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,613 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,613 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:35,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f730889[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:35,614 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:35,614 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,615 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,615 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:35,615 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:35,615 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:35,615 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:35,618 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:35,618 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,618 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,618 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:35,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:35,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:35,620 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:35,620 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:35,620 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:35,620 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:35,620 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:35,622 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,622 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,622 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,624 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,626 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,626 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,626 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:35,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:35,626 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,626 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:35,626 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,626 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,626 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:35,626 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,626 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:35,626 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,627 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:35,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490655628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:35,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490655629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,727 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:35,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:35,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:35,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:35,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:35,730 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:35,730 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,730 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:35,730 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:35,730 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,730 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,731 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,731 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,732 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,732 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:35,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:35,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:35,733 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,733 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:35,733 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:35,733 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:35,735 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,735 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,736 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,736 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:35,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:35,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba905bc[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:35,738 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:35,738 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,739 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,739 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:35,739 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:35,739 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,739 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:35,739 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:35,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:35,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,740 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,740 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:35,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:35,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:35,743 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:35,743 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:35,743 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:35,743 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:35,743 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:35,744 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:35,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:35,744 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,745 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:35,746 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:35,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,747 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:35,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:35,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:35,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:35,753 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:35,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:35,754 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:35,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:35,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:35,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:35,754 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:35,754 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:35,762 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/25148ff61ad94b29855018a57cfcade2 2023-04-03 02:56:35,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/373414e4ed3a421cb527571bd0d6ca22 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/373414e4ed3a421cb527571bd0d6ca22 2023-04-03 02:56:35,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/373414e4ed3a421cb527571bd0d6ca22, entries=350, sequenceid=392, filesize=52.9 K 2023-04-03 02:56:35,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/832782a2a1634a109eca1c6a171101cd as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/832782a2a1634a109eca1c6a171101cd 2023-04-03 02:56:35,773 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/832782a2a1634a109eca1c6a171101cd, entries=100, sequenceid=392, filesize=9.5 K 2023-04-03 02:56:35,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/25148ff61ad94b29855018a57cfcade2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/25148ff61ad94b29855018a57cfcade2 2023-04-03 02:56:35,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/25148ff61ad94b29855018a57cfcade2, entries=100, sequenceid=392, filesize=9.5 K 2023-04-03 02:56:35,777 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 1af30d70f4e72ebe50e43fde14e4fe9b in 875ms, sequenceid=392, compaction requested=true 2023-04-03 02:56:35,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:35,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:35,778 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:35,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:35,778 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:35,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:35,779 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:35,779 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/B is initiating minor compaction (all files) 2023-04-03 02:56:35,779 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/B in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,779 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/57e4a60eb1fd4834b5656e359a2eda8d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/220d437b71fe4acd9c0c43f89cd328e3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/832782a2a1634a109eca1c6a171101cd] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=34.0 K 2023-04-03 02:56:35,788 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107052 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:35,788 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/A is initiating minor compaction (all files) 2023-04-03 02:56:35,788 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/A in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,788 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6a76f7c9453347ec924657b7517e2f6f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/67a36d089e9e441c8c897c5ae67ef092, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/373414e4ed3a421cb527571bd0d6ca22] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=104.5 K 2023-04-03 02:56:35,789 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 57e4a60eb1fd4834b5656e359a2eda8d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680490594146 2023-04-03 02:56:35,790 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 220d437b71fe4acd9c0c43f89cd328e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680490594248 2023-04-03 02:56:35,790 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 6a76f7c9453347ec924657b7517e2f6f, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680490594146 2023-04-03 02:56:35,790 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 67a36d089e9e441c8c897c5ae67ef092, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680490594248 2023-04-03 02:56:35,791 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 832782a2a1634a109eca1c6a171101cd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680490594897 2023-04-03 02:56:35,791 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 373414e4ed3a421cb527571bd0d6ca22, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680490594880 2023-04-03 02:56:35,821 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#B#compaction#620 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:35,845 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#A#compaction#619 average throughput is 0.62 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:35,854 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:35,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:35,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:35,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:35,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:35,868 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:35,868 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,868 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:35,868 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:35,868 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,869 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,870 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:35,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:35,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:35,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:35,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:35,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:35,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:35,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:35,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:35,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:35,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:35,871 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:35,871 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:35,871 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:35,871 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:35,871 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:35,871 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:35,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:35,872 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-04-03 02:56:35,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:35,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:35,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:35,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:35,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:35,872 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:35,892 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403b8fa3595469f42d4b92b31051644ebd7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b8fa3595469f42d4b92b31051644ebd7 2023-04-03 02:56:35,897 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=429, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/4a53859e3a6941569076eaf7c1537e57 2023-04-03 02:56:35,908 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/2210235840b94a32a10dc8f600d50cf2 2023-04-03 02:56:36,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:36,137 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:36,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:36,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490656150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:36,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490656151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:36,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490656255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:36,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490656256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,261 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/591fee175bc6476993fe228b53040e38 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/591fee175bc6476993fe228b53040e38 2023-04-03 02:56:36,266 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/B of 1af30d70f4e72ebe50e43fde14e4fe9b into 591fee175bc6476993fe228b53040e38(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:36,266 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:36,266 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/B, priority=13, startTime=1680490595778; duration=0sec 2023-04-03 02:56:36,266 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:36,266 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:36,269 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:36,269 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/C is initiating minor compaction (all files) 2023-04-03 02:56:36,269 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/C in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:36,269 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/ef50a5f70fff4b7e8b0f84fe61cb4d0e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7fe5b47aafa244bf83a2abd1d51dd1aa, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/25148ff61ad94b29855018a57cfcade2] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=34.0 K 2023-04-03 02:56:36,269 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ef50a5f70fff4b7e8b0f84fe61cb4d0e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680490594146 2023-04-03 02:56:36,270 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7fe5b47aafa244bf83a2abd1d51dd1aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680490594248 2023-04-03 02:56:36,270 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 25148ff61ad94b29855018a57cfcade2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680490594897 2023-04-03 02:56:36,277 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#C#compaction#624 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:36,324 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/7e2e38c47a2b4a92bb9e230188eb3a24 2023-04-03 02:56:36,331 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/4a53859e3a6941569076eaf7c1537e57 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4a53859e3a6941569076eaf7c1537e57 2023-04-03 02:56:36,333 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/12af446c9c7540148d58ff5a40825f0a as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/12af446c9c7540148d58ff5a40825f0a 2023-04-03 02:56:36,335 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4a53859e3a6941569076eaf7c1537e57, entries=150, sequenceid=429, filesize=25.4 K 2023-04-03 02:56:36,336 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/2210235840b94a32a10dc8f600d50cf2 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2210235840b94a32a10dc8f600d50cf2 2023-04-03 02:56:36,337 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/67aae3b8be8841ca9c28a8606c0132ad as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/67aae3b8be8841ca9c28a8606c0132ad 2023-04-03 02:56:36,338 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/A of 1af30d70f4e72ebe50e43fde14e4fe9b into 12af446c9c7540148d58ff5a40825f0a(size=26.3 K), total size for store is 51.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:36,338 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:36,338 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/A, priority=13, startTime=1680490595778; duration=0sec 2023-04-03 02:56:36,339 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:36,342 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/C of 1af30d70f4e72ebe50e43fde14e4fe9b into 67aae3b8be8841ca9c28a8606c0132ad(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:36,342 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:36,342 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/C, priority=13, startTime=1680490595778; duration=0sec 2023-04-03 02:56:36,343 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:36,344 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2210235840b94a32a10dc8f600d50cf2, entries=150, sequenceid=429, filesize=11.9 K 2023-04-03 02:56:36,346 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/7e2e38c47a2b4a92bb9e230188eb3a24 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7e2e38c47a2b4a92bb9e230188eb3a24 2023-04-03 02:56:36,350 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7e2e38c47a2b4a92bb9e230188eb3a24, entries=150, sequenceid=429, filesize=11.9 K 2023-04-03 02:56:36,351 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=53.67 KB/54960 for 1af30d70f4e72ebe50e43fde14e4fe9b in 479ms, sequenceid=429, compaction requested=false 2023-04-03 02:56:36,351 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:36,351 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:36,351 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:36,351 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:36,351 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,351 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:36,351 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:36,353 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,353 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,354 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,354 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73ae0fef[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:36,355 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:36,355 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,357 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,357 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:36,357 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:36,357 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:36,357 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,357 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:36,359 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,359 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:36,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,359 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:36,359 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:36,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:36,363 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:36,363 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:36,363 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:36,363 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:36,363 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:36,365 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,365 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,365 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:36,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,367 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,368 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,372 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:36,372 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,372 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,372 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,372 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:36,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:36,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:36,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:36,372 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:36,372 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,372 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,372 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,372 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:36,372 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:36,372 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:36,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:36,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:36,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:36,463 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 02:56:36,463 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:36,463 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:36,463 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:36,463 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:36,464 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:36,464 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:36,476 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:36,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:36,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:36,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:36,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:36,487 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:36,487 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:36,487 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:36,487 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:36,488 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,488 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,489 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:36,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,495 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,495 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:36,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:36,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:36,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:36,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:36,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:36,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:36,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:36,496 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:36,496 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:36,496 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:36,496 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:36,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:36,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:36,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:36,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:36,499 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403c1ce54c5e0cb4b27bef8a2eba4800007 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c1ce54c5e0cb4b27bef8a2eba4800007 2023-04-03 02:56:36,499 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,499 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,500 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,500 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:36,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57307de0[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:36,503 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:36,503 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:36,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490656503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,504 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,504 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:36,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:36,505 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:36,505 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,505 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:36,507 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:36,507 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:36,507 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,507 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:36,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:36,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490656504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:36,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:36,510 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:36,510 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:36,510 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:36,510 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:36,510 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:36,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,511 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:36,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,513 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,514 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,517 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,518 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:36,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:36,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:36,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:36,518 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:36,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:36,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:36,518 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:36,518 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:36,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:36,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:36,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:36,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490656605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:36,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490656608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,618 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:36,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:36,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:36,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:36,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:36,621 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:36,621 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:36,621 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:36,621 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:36,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,622 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,623 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:36,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:36,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:36,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:36,624 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:36,624 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:36,625 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:36,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:36,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:36,628 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:36,628 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:36,628 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:36,628 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:36,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:36,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:36,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:36,628 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:36,629 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,630 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:36,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@306010cf[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:36,635 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:36,635 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,640 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,640 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,640 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:36,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,640 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:36,640 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:36,641 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:36,643 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:36,643 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:36,643 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:36,643 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:36,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:36,646 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:36,646 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:36,646 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:36,646 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:36,646 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:36,647 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,648 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,649 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,650 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,650 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:36,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:36,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:36,656 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:36,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:36,656 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,656 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,656 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,656 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,656 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,656 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,656 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:36,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:36,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:36,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:36,656 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:36,657 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:36,657 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:36,757 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:36,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:36,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:36,763 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:36,763 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:36,763 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:36,763 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:36,764 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,764 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:36,766 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:36,767 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,767 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:36,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:36,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:36,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:36,768 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:36,768 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:36,768 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:36,769 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:36,769 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:36,769 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:36,769 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,769 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:36,769 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:36,770 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,770 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,770 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,770 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:36,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23f7679a[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:36,772 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:36,772 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,773 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,774 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:36,774 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:36,774 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,774 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:36,774 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:36,775 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,775 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:36,775 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,775 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:36,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:36,777 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:36,777 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:36,778 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:36,778 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:36,778 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:36,780 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,780 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,780 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:36,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:36,781 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,782 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,784 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:36,784 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,784 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,784 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,784 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:36,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:36,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:36,784 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:36,784 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,784 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,784 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,784 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:36,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:36,784 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:36,784 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:36,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:36,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:36,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490656808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:36,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490656811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,885 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:36,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:36,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:36,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:36,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:36,888 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:36,888 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:36,888 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:36,888 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:36,888 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,888 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,889 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:36,889 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:36,890 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,890 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:36,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:36,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:36,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:36,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:36,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:36,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:36,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:36,891 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:36,891 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:36,891 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:36,891 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:36,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:36,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:36,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,891 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:36,892 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:36,893 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,893 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,893 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,893 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:36,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a5d57c[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:36,895 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:36,895 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,897 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,897 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:36,897 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,897 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:36,897 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:36,897 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:36,898 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,899 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:36,899 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,899 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:36,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:36,901 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:36,901 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:36,901 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:36,901 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:36,901 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:36,902 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,902 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:36,902 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:36,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:36,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:36,903 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,904 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:36,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:36,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,906 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,906 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,906 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:36,906 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:36,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:36,906 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:36,906 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:36,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:36,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:36,907 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:36,907 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,907 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:36,907 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:36,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:36,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:36,907 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:36,907 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:36,910 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=445, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/45c708f1e69c4058b3d06c84ae12b7b1 2023-04-03 02:56:37,007 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:37,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:37,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:37,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:37,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:37,010 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:37,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:37,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:37,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:37,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,011 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,013 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,013 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:37,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:37,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:37,013 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,013 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:37,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:37,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:37,015 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:37,015 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:37,015 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:37,015 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:37,015 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:37,016 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,016 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:37,017 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:37,017 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:37,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:37,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@742e73f5[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:37,018 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:37,018 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,019 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,020 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:37,020 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:37,020 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:37,020 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,020 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:37,021 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,021 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:37,021 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:37,021 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:37,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:37,024 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:37,024 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:37,024 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:37,024 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:37,024 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:37,026 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,026 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:37,026 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:37,026 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:37,028 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:37,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,030 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,030 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,030 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,030 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:37,030 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:37,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:37,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:37,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:37,030 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:37,030 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:37,030 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:37,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:37,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:37,030 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:37,030 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:37,030 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,030 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:37,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490657112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:37,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490657114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,130 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:37,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:37,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:37,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:37,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:37,134 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:37,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:37,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:37,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:37,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,134 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,135 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:37,135 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:37,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:37,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,136 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,136 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:37,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:37,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:37,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:37,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:37,137 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:37,137 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:37,137 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:37,137 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:37,137 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:37,138 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:37,138 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:37,138 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:37,138 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:37,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:37,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:37,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:37,138 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:37,143 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,143 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:37,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:37,143 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,143 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:37,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:37,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4478f3d7[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:37,145 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:37,145 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,146 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,146 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:37,146 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:37,147 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:37,147 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,147 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:37,148 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,148 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:37,148 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:37,148 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:37,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:37,150 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:37,150 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:37,150 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:37,150 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:37,150 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:37,151 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,151 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,151 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:37,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:37,152 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:37,153 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,156 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:37,156 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,156 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:37,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:37,156 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:37,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:37,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:37,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:37,156 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,156 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,156 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,156 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:37,156 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:37,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:37,156 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,156 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,156 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,256 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:37,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:37,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:37,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:37,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:37,260 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:37,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:37,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:37,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:37,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,260 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,261 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:37,261 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:37,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:37,262 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,262 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:37,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:37,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:37,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:37,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:37,263 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:37,263 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:37,263 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:37,263 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:37,263 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:37,265 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,265 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:37,265 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,265 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:37,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:37,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:37,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:37,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cab9775[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:37,267 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:37,267 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,268 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,268 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,268 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:37,271 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,271 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:37,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:37,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:37,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:37,273 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:37,273 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:37,273 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:37,273 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:37,273 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:37,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:37,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,275 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:37,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:37,275 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:37,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,276 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,278 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,278 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:37,278 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,278 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:37,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:37,278 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:37,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:37,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:37,278 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:37,278 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,279 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:37,279 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:37,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:37,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:37,279 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,319 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/e41a4708c1fe4bfea56d057fc8da044e 2023-04-03 02:56:37,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/6daf5f1bb08345c7bd99d2c8ebe7a4cf 2023-04-03 02:56:37,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/45c708f1e69c4058b3d06c84ae12b7b1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/45c708f1e69c4058b3d06c84ae12b7b1 2023-04-03 02:56:37,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/45c708f1e69c4058b3d06c84ae12b7b1, entries=150, sequenceid=445, filesize=25.4 K 2023-04-03 02:56:37,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/e41a4708c1fe4bfea56d057fc8da044e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e41a4708c1fe4bfea56d057fc8da044e 2023-04-03 02:56:37,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e41a4708c1fe4bfea56d057fc8da044e, entries=150, sequenceid=445, filesize=11.9 K 2023-04-03 02:56:37,339 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/6daf5f1bb08345c7bd99d2c8ebe7a4cf as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/6daf5f1bb08345c7bd99d2c8ebe7a4cf 2023-04-03 02:56:37,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/6daf5f1bb08345c7bd99d2c8ebe7a4cf, entries=150, sequenceid=445, filesize=11.9 K 2023-04-03 02:56:37,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1af30d70f4e72ebe50e43fde14e4fe9b in 879ms, sequenceid=445, compaction requested=true 2023-04-03 02:56:37,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:37,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:37,342 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:37,343 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:37,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:37,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:37,344 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78990 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:37,344 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:37,344 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/A is initiating minor compaction (all files) 2023-04-03 02:56:37,344 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/B is initiating minor compaction (all files) 2023-04-03 02:56:37,344 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/A in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:37,344 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/B in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:37,344 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/12af446c9c7540148d58ff5a40825f0a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4a53859e3a6941569076eaf7c1537e57, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/45c708f1e69c4058b3d06c84ae12b7b1] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=77.1 K 2023-04-03 02:56:37,344 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/591fee175bc6476993fe228b53040e38, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2210235840b94a32a10dc8f600d50cf2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e41a4708c1fe4bfea56d057fc8da044e] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=36.5 K 2023-04-03 02:56:37,344 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 12af446c9c7540148d58ff5a40825f0a, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680490594252 2023-04-03 02:56:37,345 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 591fee175bc6476993fe228b53040e38, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680490594252 2023-04-03 02:56:37,345 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 4a53859e3a6941569076eaf7c1537e57, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680490595007 2023-04-03 02:56:37,345 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 2210235840b94a32a10dc8f600d50cf2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680490595007 2023-04-03 02:56:37,345 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 45c708f1e69c4058b3d06c84ae12b7b1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1680490596142 2023-04-03 02:56:37,345 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting e41a4708c1fe4bfea56d057fc8da044e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1680490596142 2023-04-03 02:56:37,351 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#B#compaction#629 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:37,353 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#A#compaction#628 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:37,359 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/445b5f8f3c424912bd7657f9c5f33ad0 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/445b5f8f3c424912bd7657f9c5f33ad0 2023-04-03 02:56:37,363 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/B of 1af30d70f4e72ebe50e43fde14e4fe9b into 445b5f8f3c424912bd7657f9c5f33ad0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:37,363 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:37,363 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/B, priority=13, startTime=1680490597342; duration=0sec 2023-04-03 02:56:37,363 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:37,363 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:37,364 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:37,364 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/C is initiating minor compaction (all files) 2023-04-03 02:56:37,364 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/C in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:37,364 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/67aae3b8be8841ca9c28a8606c0132ad, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7e2e38c47a2b4a92bb9e230188eb3a24, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/6daf5f1bb08345c7bd99d2c8ebe7a4cf] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=36.5 K 2023-04-03 02:56:37,364 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 67aae3b8be8841ca9c28a8606c0132ad, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680490594252 2023-04-03 02:56:37,365 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7e2e38c47a2b4a92bb9e230188eb3a24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680490595007 2023-04-03 02:56:37,365 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 6daf5f1bb08345c7bd99d2c8ebe7a4cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1680490596142 2023-04-03 02:56:37,369 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#C#compaction#630 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:37,379 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:37,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:37,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:37,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:37,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:37,384 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:37,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:37,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:37,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:37,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,385 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:37,386 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:37,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:37,386 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,386 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:37,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:37,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:37,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:37,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:37,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:37,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:37,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:37,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:37,387 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:37,388 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:37,388 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:37,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:37,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:37,388 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 02:56:37,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:37,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:37,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:37,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:37,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:37,388 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:37,391 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/2b6db8826e3e4280adf9be9d2db756b4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2b6db8826e3e4280adf9be9d2db756b4 2023-04-03 02:56:37,395 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/C of 1af30d70f4e72ebe50e43fde14e4fe9b into 2b6db8826e3e4280adf9be9d2db756b4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:37,395 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:37,395 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/C, priority=13, startTime=1680490597343; duration=0sec 2023-04-03 02:56:37,395 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:37,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:37,618 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:37,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:37,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490657635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:37,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490657635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:37,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490657739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:37,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490657739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,781 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/126d31d5a29641009302f73a8fdb591e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/126d31d5a29641009302f73a8fdb591e 2023-04-03 02:56:37,785 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/A of 1af30d70f4e72ebe50e43fde14e4fe9b into 126d31d5a29641009302f73a8fdb591e(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:37,785 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:37,785 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/A, priority=13, startTime=1680490597342; duration=0sec 2023-04-03 02:56:37,786 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:37,800 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304039d8e0cd2bcb64ab9a63daa674bce897c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039d8e0cd2bcb64ab9a63daa674bce897c 2023-04-03 02:56:37,808 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=465, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/bbcf346109a249c39594106161cee938 2023-04-03 02:56:37,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:37,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490657942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:37,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:37,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490657942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,224 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/fc1ecdf7e6df4758a3d6366588edf62e 2023-04-03 02:56:38,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:38,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490658246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:38,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490658246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,249 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/2aa515a4d2e9439a9c651ea82b0a8cfb 2023-04-03 02:56:38,253 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/bbcf346109a249c39594106161cee938 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/bbcf346109a249c39594106161cee938 2023-04-03 02:56:38,256 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/bbcf346109a249c39594106161cee938, entries=150, sequenceid=465, filesize=25.4 K 2023-04-03 02:56:38,257 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/fc1ecdf7e6df4758a3d6366588edf62e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/fc1ecdf7e6df4758a3d6366588edf62e 2023-04-03 02:56:38,261 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/fc1ecdf7e6df4758a3d6366588edf62e, entries=150, sequenceid=465, filesize=11.9 K 2023-04-03 02:56:38,262 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/2aa515a4d2e9439a9c651ea82b0a8cfb as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2aa515a4d2e9439a9c651ea82b0a8cfb 2023-04-03 02:56:38,267 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2aa515a4d2e9439a9c651ea82b0a8cfb, entries=150, sequenceid=465, filesize=11.9 K 2023-04-03 02:56:38,268 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1af30d70f4e72ebe50e43fde14e4fe9b in 880ms, sequenceid=465, compaction requested=false 2023-04-03 02:56:38,268 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:38,268 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:38,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:38,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:38,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:38,268 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:38,271 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,271 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,271 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,271 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:38,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:38,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52abb7a9[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:38,274 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:38,274 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,276 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,276 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:38,276 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:38,276 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:38,276 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,276 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:38,278 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,278 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:38,278 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:38,278 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:38,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:38,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:38,283 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:38,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:38,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:38,283 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:38,283 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:38,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,285 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:38,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:38,286 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:38,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,287 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:38,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:38,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:38,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:38,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:38,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:38,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:38,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:38,291 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:38,291 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,291 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:38,291 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:38,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:38,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:38,392 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:38,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:38,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:38,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:38,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:38,395 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:38,395 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:38,395 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:38,395 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:38,395 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,396 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:38,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,397 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:38,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:38,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:38,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,397 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:38,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:38,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:38,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:38,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:38,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:38,398 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:38,398 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:38,398 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:38,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:38,399 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:38,399 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:38,399 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:38,399 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 02:56:38,399 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:38,399 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:38,399 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:38,399 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:38,399 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:38,399 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:38,412 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403bc17fca84a734229b6f3bc74535be2f1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bc17fca84a734229b6f3bc74535be2f1 2023-04-03 02:56:38,419 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=480, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/b0fda944c32b4089a69791f828e84b29 2023-04-03 02:56:38,430 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/906168292d7d4ac69a57f4f6c10e8225 2023-04-03 02:56:38,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:38,751 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:38,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:38,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490658772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:38,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490658773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,840 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/b1943e790c84460980cf29d3e2216d6d 2023-04-03 02:56:38,844 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/b0fda944c32b4089a69791f828e84b29 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/b0fda944c32b4089a69791f828e84b29 2023-04-03 02:56:38,847 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/b0fda944c32b4089a69791f828e84b29, entries=150, sequenceid=480, filesize=25.4 K 2023-04-03 02:56:38,847 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/906168292d7d4ac69a57f4f6c10e8225 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/906168292d7d4ac69a57f4f6c10e8225 2023-04-03 02:56:38,850 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/906168292d7d4ac69a57f4f6c10e8225, entries=150, sequenceid=480, filesize=11.9 K 2023-04-03 02:56:38,851 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/b1943e790c84460980cf29d3e2216d6d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b1943e790c84460980cf29d3e2216d6d 2023-04-03 02:56:38,854 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b1943e790c84460980cf29d3e2216d6d, entries=150, sequenceid=480, filesize=11.9 K 2023-04-03 02:56:38,854 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1af30d70f4e72ebe50e43fde14e4fe9b in 455ms, sequenceid=480, compaction requested=true 2023-04-03 02:56:38,854 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:38,854 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:38,854 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:38,855 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:38,855 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,855 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:38,855 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:38,856 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,856 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:38,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,857 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:38,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:38,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@551dc7d2[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:38,858 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:38,858 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,866 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,866 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:38,866 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:38,866 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:38,866 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:38,868 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,868 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:38,868 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:38,868 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:38,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:38,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:38,870 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:38,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:38,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:38,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:38,870 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:38,872 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,872 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:38,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,872 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:38,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:38,874 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:38,875 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:38,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 02:56:38,878 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:38,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:38,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:38,878 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:38,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:38,878 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,878 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:38,878 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:38,878 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,878 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,878 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,878 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:38,878 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:38,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:38,878 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:38,878 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:38,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:38,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:38,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:38,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:38,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:38,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:38,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490658890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:38,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490658890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,908 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304038d0248ca17ce429eb8db8c34eb86eeed to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038d0248ca17ce429eb8db8c34eb86eeed 2023-04-03 02:56:38,978 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:38,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:38,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:38,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:38,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:38,981 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:38,981 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:38,981 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:38,981 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:38,982 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,982 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:38,983 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:38,983 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:38,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:38,983 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,983 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:38,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:38,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:38,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:38,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:38,985 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:38,985 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:38,985 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:38,985 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:38,985 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:38,985 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:38,985 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:38,985 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:38,985 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:38,985 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:38,985 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:38,985 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,985 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:38,985 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:38,987 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:38,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:38,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:38,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:38,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56f870ed[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:38,989 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:38,989 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,990 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:38,990 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:38,990 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:38,990 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:38,990 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,990 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:38,991 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:38,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:38,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:38,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:38,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:38,993 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:38,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:38,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:38,994 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:38,994 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:38,995 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,995 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:38,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:38,995 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:38,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:38,995 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:38,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:38,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:38,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:38,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,996 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:38,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490658994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:39,000 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:39,000 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,000 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,001 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490658997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:39,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:39,001 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,001 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,001 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,001 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:39,001 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:39,001 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,001 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,001 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,101 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:39,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:39,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:39,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:39,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:39,110 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:39,110 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,110 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:39,110 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:39,110 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,110 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,111 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,111 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,112 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,112 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:39,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:39,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:39,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:39,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:39,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:39,113 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,113 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:39,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:39,113 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:39,113 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:39,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:39,113 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,114 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:39,114 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:39,115 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,115 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,115 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,115 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:39,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56684578[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:39,116 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:39,116 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,118 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:39,118 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,118 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:39,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:39,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:39,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:39,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:39,123 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:39,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:39,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:39,123 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:39,123 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:39,124 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,124 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,125 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,125 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,126 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,128 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,128 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,128 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,128 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:39,128 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:39,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:39,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,129 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:39,129 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:39,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:39,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490659198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:39,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490659201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,229 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:39,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:39,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:39,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:39,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:39,232 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:39,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:39,232 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:39,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,233 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,234 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,234 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,234 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,234 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:39,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:39,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:39,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:39,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:39,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:39,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:39,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:39,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:39,236 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,236 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:39,236 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:39,236 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:39,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:39,236 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,237 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:39,237 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:39,239 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,239 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,240 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:39,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f1a6e3[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:39,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:39,241 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,242 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:39,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,242 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:39,250 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:39,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,250 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:39,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:39,253 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:39,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:39,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:39,253 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:39,253 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:39,255 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,255 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,256 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,256 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,257 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,259 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:39,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:39,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,259 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,260 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:39,260 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:39,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:39,313 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=500, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/562cc14f7ac149789e8668b102dde4c3 2023-04-03 02:56:39,360 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:39,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:39,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:39,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:39,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:39,363 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:39,363 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,363 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:39,363 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:39,364 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,364 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,365 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,365 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:39,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:39,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:39,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:39,366 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:39,366 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,366 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:39,366 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:39,366 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:39,366 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,366 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:39,366 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:39,367 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,367 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:39,367 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:39,367 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,367 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:39,367 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:39,368 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,368 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:39,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:39,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62e9c958[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:39,370 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:39,370 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,377 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:39,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:39,377 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:39,377 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,377 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:39,378 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:39,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,378 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:39,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:39,381 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:39,381 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:39,381 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:39,381 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:39,381 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:39,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,382 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,383 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,384 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:39,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:39,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,386 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,386 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:39,386 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:39,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,386 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:39,387 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,487 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:39,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:39,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:39,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:39,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:39,490 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:39,490 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,490 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:39,490 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:39,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,491 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,493 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,493 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:39,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:39,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:39,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:39,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:39,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:39,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,494 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:39,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:39,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:39,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:39,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:39,495 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:39,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:39,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:39,495 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:39,496 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,496 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,497 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,497 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:39,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:39,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7783b2f0[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:39,498 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:39,498 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,499 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,499 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:39,499 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:39,499 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:39,499 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,499 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:39,500 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,500 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490659501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:39,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:39,503 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:39,503 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:39,503 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:39,503 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:39,503 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:39,506 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,506 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,506 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,506 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,507 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490659505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,509 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,509 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,509 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,509 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,509 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:39,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:39,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,510 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,510 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:39,510 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:39,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:39,509 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,509 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,510 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,510 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,610 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:39,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:39,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:39,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:39,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:39,613 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:39,613 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,613 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:39,614 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:39,614 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,614 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,615 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,615 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,615 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,616 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:39,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:39,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:39,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:39,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:39,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:39,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:39,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:39,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:39,617 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,617 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:39,617 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:39,617 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:39,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:39,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:39,617 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:39,619 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,619 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,619 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,619 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:39,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63d46afd[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:39,620 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:39,621 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,623 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:39,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:39,623 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:39,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,623 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:39,624 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,624 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:39,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:39,627 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:39,627 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:39,627 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:39,627 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:39,627 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:39,628 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,628 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,628 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,629 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,630 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,635 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,635 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,635 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:39,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:39,635 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,636 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,636 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:39,636 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:39,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,636 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,636 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,636 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,636 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,637 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/ffd84227750e4af6b10478e098356996 2023-04-03 02:56:39,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/9a0736034c3546698df3f19dc1e87224 2023-04-03 02:56:39,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/562cc14f7ac149789e8668b102dde4c3 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/562cc14f7ac149789e8668b102dde4c3 2023-04-03 02:56:39,736 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:39,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/562cc14f7ac149789e8668b102dde4c3, entries=150, sequenceid=500, filesize=25.4 K 2023-04-03 02:56:39,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:39,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/ffd84227750e4af6b10478e098356996 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ffd84227750e4af6b10478e098356996 2023-04-03 02:56:39,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:39,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:39,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:39,739 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:39,739 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,739 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:39,739 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:39,740 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,740 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ffd84227750e4af6b10478e098356996, entries=150, sequenceid=500, filesize=11.9 K 2023-04-03 02:56:39,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/9a0736034c3546698df3f19dc1e87224 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/9a0736034c3546698df3f19dc1e87224 2023-04-03 02:56:39,743 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,744 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:39,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:39,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:39,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:39,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:39,746 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/9a0736034c3546698df3f19dc1e87224, entries=150, sequenceid=500, filesize=11.9 K 2023-04-03 02:56:39,746 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,746 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:39,746 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:39,746 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:39,747 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1af30d70f4e72ebe50e43fde14e4fe9b in 870ms, sequenceid=500, compaction requested=true 2023-04-03 02:56:39,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:39,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:39,747 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:39,747 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:39,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:39,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:39,748 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,748 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,748 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,748 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,749 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105138 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,749 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/A is initiating minor compaction (all files) 2023-04-03 02:56:39,749 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/A in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,749 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/126d31d5a29641009302f73a8fdb591e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/bbcf346109a249c39594106161cee938, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/b0fda944c32b4089a69791f828e84b29, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/562cc14f7ac149789e8668b102dde4c3] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=102.7 K 2023-04-03 02:56:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,749 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:39,749 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 126d31d5a29641009302f73a8fdb591e, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1680490596142 2023-04-03 02:56:39,749 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/B is initiating minor compaction (all files) 2023-04-03 02:56:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,749 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/B in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,749 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/445b5f8f3c424912bd7657f9c5f33ad0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/fc1ecdf7e6df4758a3d6366588edf62e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/906168292d7d4ac69a57f4f6c10e8225, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ffd84227750e4af6b10478e098356996] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=48.5 K 2023-04-03 02:56:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,750 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting bbcf346109a249c39594106161cee938, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680490596491 2023-04-03 02:56:39,750 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 445b5f8f3c424912bd7657f9c5f33ad0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1680490596142 2023-04-03 02:56:39,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:39,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37460eda[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:39,750 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:39,750 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,750 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting fc1ecdf7e6df4758a3d6366588edf62e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680490596491 2023-04-03 02:56:39,750 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting b0fda944c32b4089a69791f828e84b29, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680490597631 2023-04-03 02:56:39,751 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 906168292d7d4ac69a57f4f6c10e8225, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680490597631 2023-04-03 02:56:39,751 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 562cc14f7ac149789e8668b102dde4c3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680490598759 2023-04-03 02:56:39,751 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting ffd84227750e4af6b10478e098356996, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680490598759 2023-04-03 02:56:39,751 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,751 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:39,752 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:39,755 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:39,755 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,755 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:39,755 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:39,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:39,758 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:39,758 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:39,758 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:39,758 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:39,758 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:39,760 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,760 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:39,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,760 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,761 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#A#compaction#640 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:39,762 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,762 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#B#compaction#641 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,763 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,765 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:39,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,766 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:39,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:39,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:39,766 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:39,766 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:39,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:39,766 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:39,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:39,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:39,778 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/a784ac663c4c4de7912758a39dba703e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/a784ac663c4c4de7912758a39dba703e 2023-04-03 02:56:39,782 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/B of 1af30d70f4e72ebe50e43fde14e4fe9b into a784ac663c4c4de7912758a39dba703e(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:39,782 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:39,782 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/B, priority=12, startTime=1680490599747; duration=0sec 2023-04-03 02:56:39,782 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:39,782 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 02:56:39,783 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 02:56:39,783 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/C is initiating minor compaction (all files) 2023-04-03 02:56:39,783 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/C in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,783 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2b6db8826e3e4280adf9be9d2db756b4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2aa515a4d2e9439a9c651ea82b0a8cfb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b1943e790c84460980cf29d3e2216d6d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/9a0736034c3546698df3f19dc1e87224] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=48.5 K 2023-04-03 02:56:39,784 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 2b6db8826e3e4280adf9be9d2db756b4, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1680490596142 2023-04-03 02:56:39,784 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 2aa515a4d2e9439a9c651ea82b0a8cfb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680490596491 2023-04-03 02:56:39,784 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting b1943e790c84460980cf29d3e2216d6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680490597631 2023-04-03 02:56:39,784 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 9a0736034c3546698df3f19dc1e87224, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680490598759 2023-04-03 02:56:39,791 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#C#compaction#642 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:39,866 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:39,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:39,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 02:56:39,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 02:56:39,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 02:56:39,869 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 02:56:39,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 02:56:39,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 02:56:39,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,870 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,871 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:39,871 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:39,872 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:39,872 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 02:56:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 02:56:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:39,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 02:56:39,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 02:56:39,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 02:56:39,873 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 02:56:39,873 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 02:56:39,873 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 02:56:39,873 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 02:56:39,873 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 02:56:39,873 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:39,873 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. started... 2023-04-03 02:56:39,873 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 02:56:39,873 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:39,873 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:39,873 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:39,874 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:39,874 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:39,874 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:39,884 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304036a26bf0d05924abc8e5dc4748ff748ce to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036a26bf0d05924abc8e5dc4748ff748ce 2023-04-03 02:56:40,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:40,007 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. as already flushing 2023-04-03 02:56:40,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:40,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490660030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:40,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:40,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490660031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:40,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:40,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43815] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490660133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:40,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:40,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490660134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:40,186 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/caa9877a455d40848f361b7eb91ebbda as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/caa9877a455d40848f361b7eb91ebbda 2023-04-03 02:56:40,190 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/A of 1af30d70f4e72ebe50e43fde14e4fe9b into caa9877a455d40848f361b7eb91ebbda(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:40,190 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:40,190 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/A, priority=12, startTime=1680490599747; duration=0sec 2023-04-03 02:56:40,190 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:40,205 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/eaa18c52b13b49078b8e2ca16b7d51cb as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/eaa18c52b13b49078b8e2ca16b7d51cb 2023-04-03 02:56:40,208 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/C of 1af30d70f4e72ebe50e43fde14e4fe9b into eaa18c52b13b49078b8e2ca16b7d51cb(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:40,208 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:40,208 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/C, priority=12, startTime=1680490599747; duration=0sec 2023-04-03 02:56:40,208 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:40,289 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=513, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/4103a25b6ca54d478dca5d31e1b9f84b 2023-04-03 02:56:40,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:40,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43815] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490660336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:40,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:40,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490660336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:40,596 DEBUG [Thread-3917] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e6a6f3c to 127.0.0.1:52506 2023-04-03 02:56:40,596 DEBUG [Thread-3917] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:40,597 DEBUG [Thread-3923] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3dca23c0 to 127.0.0.1:52506 2023-04-03 02:56:40,597 DEBUG [Thread-3923] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:40,598 DEBUG [Thread-3919] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a470a3d to 127.0.0.1:52506 2023-04-03 02:56:40,598 DEBUG [Thread-3919] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:40,598 DEBUG [Thread-3915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:52506 2023-04-03 02:56:40,598 DEBUG [Thread-3915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:40,599 DEBUG [Thread-3921] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bb72437 to 127.0.0.1:52506 2023-04-03 02:56:40,600 DEBUG [Thread-3921] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:40,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:40,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46108 deadline: 1680490660638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:40,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 02:56:40,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46142 deadline: 1680490660640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:40,697 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/72416bac7cb04390971671ed50912ce1 2023-04-03 02:56:41,106 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/5edebd48ffb24c19a080052d75933b87 2023-04-03 02:56:41,109 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/4103a25b6ca54d478dca5d31e1b9f84b as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4103a25b6ca54d478dca5d31e1b9f84b 2023-04-03 02:56:41,112 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4103a25b6ca54d478dca5d31e1b9f84b, entries=150, sequenceid=513, filesize=25.4 K 2023-04-03 02:56:41,112 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/72416bac7cb04390971671ed50912ce1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/72416bac7cb04390971671ed50912ce1 2023-04-03 02:56:41,115 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/72416bac7cb04390971671ed50912ce1, entries=150, sequenceid=513, filesize=11.9 K 2023-04-03 02:56:41,115 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/5edebd48ffb24c19a080052d75933b87 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5edebd48ffb24c19a080052d75933b87 2023-04-03 02:56:41,118 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5edebd48ffb24c19a080052d75933b87, entries=150, sequenceid=513, filesize=11.9 K 2023-04-03 02:56:41,119 INFO [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1af30d70f4e72ebe50e43fde14e4fe9b in 1246ms, sequenceid=513, compaction requested=false 2023-04-03 02:56:41,119 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:41,119 DEBUG [rs(jenkins-hbase4.apache.org,43815,1680490421348)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:41,119 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 02:56:41,119 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 02:56:41,119 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,119 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 02:56:41,119 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 02:56:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:41,120 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:41,121 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 02:56:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:41,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43815,1680490421348' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 02:56:41,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b49b263[Count = 0] remaining members to acquire global barrier 2023-04-03 02:56:41,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 02:56:41,122 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:41,124 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:41,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:41,124 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 02:56:41,124 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 02:56:41,124 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,43815,1680490421348' in zk 2023-04-03 02:56:41,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,124 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 02:56:41,125 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,125 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 02:56:41,125 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:41,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:41,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:41,125 DEBUG [member: 'jenkins-hbase4.apache.org,43815,1680490421348' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 02:56:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,43815,1680490421348': 2023-04-03 02:56:41,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43815,1680490421348' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 02:56:41,127 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 02:56:41,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 02:56:41,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 02:56:41,127 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 02:56:41,127 INFO [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 02:56:41,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:41,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:41,129 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 02:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 02:56:41,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:41,129 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 02:56:41,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 02:56:41,130 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 02:56:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 02:56:41,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,131 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,131 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 02:56:41,131 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:41,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 02:56:41,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 02:56:41,132 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 02:56:41,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 02:56:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 02:56:41,132 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 02:56:41,132 DEBUG [(jenkins-hbase4.apache.org,37975,1680490420046)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 02:56:41,132 DEBUG [Thread-3914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 02:56:41,132 DEBUG [Thread-3914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 02:56:41,132 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:41,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 02:56:41,132 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:41,132 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 02:56:41,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 02:56:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 02:56:41,132 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 02:56:41,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43815] regionserver.HRegion(9158): Flush requested on 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:41,141 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-03 02:56:41,141 DEBUG [Thread-3910] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66b5b1f3 to 127.0.0.1:52506 2023-04-03 02:56:41,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:41,141 DEBUG [Thread-3910] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:41,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:41,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:41,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:41,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:41,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 02:56:41,143 DEBUG [Thread-3906] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59185ba7 to 127.0.0.1:52506 2023-04-03 02:56:41,143 DEBUG [Thread-3906] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:41,232 DEBUG [Thread-3914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 02:56:41,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 02:56:41,551 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403348447b4e9eb48c2ab761ea4370ce244 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403348447b4e9eb48c2ab761ea4370ce244 2023-04-03 02:56:41,555 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=536, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/aa2b426de8804eac91b2f7b4169fa12d 2023-04-03 02:56:41,962 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/432c15800fa846afb4d559048a4878b8 2023-04-03 02:56:42,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/7778f100083a4a49a9d7dd9798311618 2023-04-03 02:56:42,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/aa2b426de8804eac91b2f7b4169fa12d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/aa2b426de8804eac91b2f7b4169fa12d 2023-04-03 02:56:42,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/aa2b426de8804eac91b2f7b4169fa12d, entries=150, sequenceid=536, filesize=25.4 K 2023-04-03 02:56:42,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/432c15800fa846afb4d559048a4878b8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/432c15800fa846afb4d559048a4878b8 2023-04-03 02:56:42,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/432c15800fa846afb4d559048a4878b8, entries=150, sequenceid=536, filesize=11.9 K 2023-04-03 02:56:42,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/7778f100083a4a49a9d7dd9798311618 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7778f100083a4a49a9d7dd9798311618 2023-04-03 02:56:42,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7778f100083a4a49a9d7dd9798311618, entries=150, sequenceid=536, filesize=11.9 K 2023-04-03 02:56:42,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=6.71 KB/6870 for 1af30d70f4e72ebe50e43fde14e4fe9b in 1243ms, sequenceid=536, compaction requested=true 2023-04-03 02:56:42,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:42,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:42,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:42,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:42,384 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:42,384 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:42,385 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79228 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:42,385 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/A is initiating minor compaction (all files) 2023-04-03 02:56:42,385 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:42,385 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/A in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:42,385 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/B is initiating minor compaction (all files) 2023-04-03 02:56:42,385 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/caa9877a455d40848f361b7eb91ebbda, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4103a25b6ca54d478dca5d31e1b9f84b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/aa2b426de8804eac91b2f7b4169fa12d] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=77.4 K 2023-04-03 02:56:42,385 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/B in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:42,385 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/a784ac663c4c4de7912758a39dba703e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/72416bac7cb04390971671ed50912ce1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/432c15800fa846afb4d559048a4878b8] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=36.8 K 2023-04-03 02:56:42,385 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting caa9877a455d40848f361b7eb91ebbda, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680490598759 2023-04-03 02:56:42,385 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting a784ac663c4c4de7912758a39dba703e, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680490598759 2023-04-03 02:56:42,385 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting 4103a25b6ca54d478dca5d31e1b9f84b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1680490598881 2023-04-03 02:56:42,385 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 72416bac7cb04390971671ed50912ce1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1680490598881 2023-04-03 02:56:42,386 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] compactions.Compactor(207): Compacting aa2b426de8804eac91b2f7b4169fa12d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1680490600029 2023-04-03 02:56:42,386 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 432c15800fa846afb4d559048a4878b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1680490600029 2023-04-03 02:56:42,391 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#B#compaction#650 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:42,392 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#A#compaction#649 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 69.23 MB/second 2023-04-03 02:56:42,798 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/524843df737146d18f2cb7d8d08a2b46 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/524843df737146d18f2cb7d8d08a2b46 2023-04-03 02:56:42,802 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/B of 1af30d70f4e72ebe50e43fde14e4fe9b into 524843df737146d18f2cb7d8d08a2b46(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:42,802 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:42,802 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/B, priority=13, startTime=1680490602384; duration=0sec 2023-04-03 02:56:42,802 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 02:56:42,802 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 02:56:42,803 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 02:56:42,803 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1912): 1af30d70f4e72ebe50e43fde14e4fe9b/C is initiating minor compaction (all files) 2023-04-03 02:56:42,803 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1af30d70f4e72ebe50e43fde14e4fe9b/C in TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:42,803 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/eaa18c52b13b49078b8e2ca16b7d51cb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5edebd48ffb24c19a080052d75933b87, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7778f100083a4a49a9d7dd9798311618] into tmpdir=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp, totalSize=36.8 K 2023-04-03 02:56:42,803 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting eaa18c52b13b49078b8e2ca16b7d51cb, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680490598759 2023-04-03 02:56:42,803 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 5edebd48ffb24c19a080052d75933b87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1680490598881 2023-04-03 02:56:42,804 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] compactions.Compactor(207): Compacting 7778f100083a4a49a9d7dd9798311618, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1680490600029 2023-04-03 02:56:42,808 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] throttle.PressureAwareThroughputController(145): 1af30d70f4e72ebe50e43fde14e4fe9b#C#compaction#651 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 02:56:43,206 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/3941a9f163564843b2d28c01b6a450e4 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/3941a9f163564843b2d28c01b6a450e4 2023-04-03 02:56:43,209 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/A of 1af30d70f4e72ebe50e43fde14e4fe9b into 3941a9f163564843b2d28c01b6a450e4(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:43,209 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:43,209 INFO [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/A, priority=13, startTime=1680490602384; duration=0sec 2023-04-03 02:56:43,209 DEBUG [RS:0;jenkins-hbase4:43815-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:43,214 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/0a78215ae6884ae1b87c621461cde67d as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/0a78215ae6884ae1b87c621461cde67d 2023-04-03 02:56:43,217 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1af30d70f4e72ebe50e43fde14e4fe9b/C of 1af30d70f4e72ebe50e43fde14e4fe9b into 0a78215ae6884ae1b87c621461cde67d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 02:56:43,217 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:43,217 INFO [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b., storeName=1af30d70f4e72ebe50e43fde14e4fe9b/C, priority=13, startTime=1680490602384; duration=0sec 2023-04-03 02:56:43,217 DEBUG [RS:0;jenkins-hbase4:43815-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 02:56:45,295 DEBUG [Thread-3908] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x062b3caa to 127.0.0.1:52506 2023-04-03 02:56:45,295 DEBUG [Thread-3908] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:45,311 DEBUG [Thread-3904] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63da10ca to 127.0.0.1:52506 2023-04-03 02:56:45,311 DEBUG [Thread-3904] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:45,324 DEBUG [Thread-3912] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x544a4215 to 127.0.0.1:52506 2023-04-03 02:56:45,324 DEBUG [Thread-3912] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:45,324 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 02:56:45,324 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 40 2023-04-03 02:56:45,324 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 158 2023-04-03 02:56:45,324 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-04-03 02:56:45,324 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 157 2023-04-03 02:56:45,324 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-04-03 02:56:45,324 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 02:56:45,324 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 4903 2023-04-03 02:56:45,324 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 4801 2023-04-03 02:56:45,324 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 4921 2023-04-03 02:56:45,324 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 4901 2023-04-03 02:56:45,325 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(398): read 4840 2023-04-03 02:56:45,325 INFO [Listener at localhost/46853] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 02:56:45,325 INFO [Listener at localhost/46853] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 02:56:45,325 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03ced4ba to 127.0.0.1:52506 2023-04-03 02:56:45,325 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:45,325 INFO [Listener at localhost/46853] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 02:56:45,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 02:56:45,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:45,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 02:56:45,329 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490605328"}]},"ts":"1680490605328"} 2023-04-03 02:56:45,329 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 02:56:45,330 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 02:56:45,331 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1af30d70f4e72ebe50e43fde14e4fe9b, UNASSIGN}] 2023-04-03 02:56:45,332 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1af30d70f4e72ebe50e43fde14e4fe9b, UNASSIGN 2023-04-03 02:56:45,333 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=1af30d70f4e72ebe50e43fde14e4fe9b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:45,333 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490605333"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680490605333"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680490605333"}]},"ts":"1680490605333"} 2023-04-03 02:56:45,334 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348}] 2023-04-03 02:56:45,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 02:56:45,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:45,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1af30d70f4e72ebe50e43fde14e4fe9b, disabling compactions & flushes 2023-04-03 02:56:45,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:45,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:45,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. after waiting 0 ms 2023-04-03 02:56:45,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:45,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1af30d70f4e72ebe50e43fde14e4fe9b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 02:56:45,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=A 2023-04-03 02:56:45,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:45,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=B 2023-04-03 02:56:45,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:45,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1af30d70f4e72ebe50e43fde14e4fe9b, store=C 2023-04-03 02:56:45,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 02:56:45,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 02:56:45,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040382331f45cefd4696ad73231c9f23f2bc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040382331f45cefd4696ad73231c9f23f2bc 2023-04-03 02:56:45,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 02:56:46,300 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=546, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/d27b05a4bcac4ee6a26d32833e6fe25f 2023-04-03 02:56:46,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/c51a274560bc46568c9d22ce0dbc7705 2023-04-03 02:56:46,314 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/e8b03e3b270748b4bfb6541856ed8518 2023-04-03 02:56:46,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/A/d27b05a4bcac4ee6a26d32833e6fe25f as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/d27b05a4bcac4ee6a26d32833e6fe25f 2023-04-03 02:56:46,326 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/d27b05a4bcac4ee6a26d32833e6fe25f, entries=100, sequenceid=546, filesize=18.6 K 2023-04-03 02:56:46,326 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/B/c51a274560bc46568c9d22ce0dbc7705 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/c51a274560bc46568c9d22ce0dbc7705 2023-04-03 02:56:46,329 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/c51a274560bc46568c9d22ce0dbc7705, entries=100, sequenceid=546, filesize=9.5 K 2023-04-03 02:56:46,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/.tmp/C/e8b03e3b270748b4bfb6541856ed8518 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/e8b03e3b270748b4bfb6541856ed8518 2023-04-03 02:56:46,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/e8b03e3b270748b4bfb6541856ed8518, entries=100, sequenceid=546, filesize=9.5 K 2023-04-03 02:56:46,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 1af30d70f4e72ebe50e43fde14e4fe9b in 848ms, sequenceid=546, compaction requested=false 2023-04-03 02:56:46,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6b130abf926846d7a9caafcbecc47047, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/f62d4c76399447d18b1d178f782125b5, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/230ab9197a0641acadff77913ad64600, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6f3058aaed284559b758f32100f34a8b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/ed35b537b01946ecb11410e40e62f85a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/198846129a3549d79407310e81e61e92, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/e0d709d8a44e476bb4d1ed9ff2f5048a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/d83b2839ec5644658934155e051a2346, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5cbb43b2787b4e4c9d53298c6c7bcb4c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/851ac621215847e1a5f2c4337aeaa5f8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5362183ac0894f73b80b8e8ceb18e3ac, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/1b1955a90ec84e81bb37b8adf22337f8, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/27e0e52612a14686a6c913d19fdc65f9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/990b1c74667b4921945d3370d50f494c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6f6228baffbd4bc69659747491d80ece, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/8d2aa5abed4b4e6b8e318bb2643adf6a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/fb4b1cef68844bc2b58c3d6638178f0a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/44df0498b8344ec7965f77465f780e04, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/85af34c3d8454c1fafe755fdaf263fde, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/81e0cd4510b14286a69ba12a86f99cf6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/278d59f86c9447d2a6ae501235e7259f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/c0491d4fe9004645acf138a25d59902c, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6a76f7c9453347ec924657b7517e2f6f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/67a36d089e9e441c8c897c5ae67ef092, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/373414e4ed3a421cb527571bd0d6ca22, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/12af446c9c7540148d58ff5a40825f0a, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4a53859e3a6941569076eaf7c1537e57, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/126d31d5a29641009302f73a8fdb591e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/45c708f1e69c4058b3d06c84ae12b7b1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/bbcf346109a249c39594106161cee938, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/b0fda944c32b4089a69791f828e84b29, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/caa9877a455d40848f361b7eb91ebbda, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/562cc14f7ac149789e8668b102dde4c3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4103a25b6ca54d478dca5d31e1b9f84b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/aa2b426de8804eac91b2f7b4169fa12d] to archive 2023-04-03 02:56:46,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:56:46,336 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6b130abf926846d7a9caafcbecc47047 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6b130abf926846d7a9caafcbecc47047 2023-04-03 02:56:46,336 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/f62d4c76399447d18b1d178f782125b5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/f62d4c76399447d18b1d178f782125b5 2023-04-03 02:56:46,337 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/230ab9197a0641acadff77913ad64600 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/230ab9197a0641acadff77913ad64600 2023-04-03 02:56:46,338 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6f3058aaed284559b758f32100f34a8b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6f3058aaed284559b758f32100f34a8b 2023-04-03 02:56:46,339 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/ed35b537b01946ecb11410e40e62f85a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/ed35b537b01946ecb11410e40e62f85a 2023-04-03 02:56:46,340 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/198846129a3549d79407310e81e61e92 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/198846129a3549d79407310e81e61e92 2023-04-03 02:56:46,340 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/e0d709d8a44e476bb4d1ed9ff2f5048a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/e0d709d8a44e476bb4d1ed9ff2f5048a 2023-04-03 02:56:46,341 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/d83b2839ec5644658934155e051a2346 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/d83b2839ec5644658934155e051a2346 2023-04-03 02:56:46,342 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5cbb43b2787b4e4c9d53298c6c7bcb4c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5cbb43b2787b4e4c9d53298c6c7bcb4c 2023-04-03 02:56:46,343 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/851ac621215847e1a5f2c4337aeaa5f8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/851ac621215847e1a5f2c4337aeaa5f8 2023-04-03 02:56:46,344 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5362183ac0894f73b80b8e8ceb18e3ac to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/5362183ac0894f73b80b8e8ceb18e3ac 2023-04-03 02:56:46,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/1b1955a90ec84e81bb37b8adf22337f8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/1b1955a90ec84e81bb37b8adf22337f8 2023-04-03 02:56:46,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/27e0e52612a14686a6c913d19fdc65f9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/27e0e52612a14686a6c913d19fdc65f9 2023-04-03 02:56:46,346 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/990b1c74667b4921945d3370d50f494c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/990b1c74667b4921945d3370d50f494c 2023-04-03 02:56:46,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6f6228baffbd4bc69659747491d80ece to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6f6228baffbd4bc69659747491d80ece 2023-04-03 02:56:46,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/8d2aa5abed4b4e6b8e318bb2643adf6a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/8d2aa5abed4b4e6b8e318bb2643adf6a 2023-04-03 02:56:46,349 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/fb4b1cef68844bc2b58c3d6638178f0a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/fb4b1cef68844bc2b58c3d6638178f0a 2023-04-03 02:56:46,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/44df0498b8344ec7965f77465f780e04 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/44df0498b8344ec7965f77465f780e04 2023-04-03 02:56:46,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/85af34c3d8454c1fafe755fdaf263fde to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/85af34c3d8454c1fafe755fdaf263fde 2023-04-03 02:56:46,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/81e0cd4510b14286a69ba12a86f99cf6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/81e0cd4510b14286a69ba12a86f99cf6 2023-04-03 02:56:46,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/278d59f86c9447d2a6ae501235e7259f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/278d59f86c9447d2a6ae501235e7259f 2023-04-03 02:56:46,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/c0491d4fe9004645acf138a25d59902c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/c0491d4fe9004645acf138a25d59902c 2023-04-03 02:56:46,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6a76f7c9453347ec924657b7517e2f6f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/6a76f7c9453347ec924657b7517e2f6f 2023-04-03 02:56:46,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/67a36d089e9e441c8c897c5ae67ef092 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/67a36d089e9e441c8c897c5ae67ef092 2023-04-03 02:56:46,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/373414e4ed3a421cb527571bd0d6ca22 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/373414e4ed3a421cb527571bd0d6ca22 2023-04-03 02:56:46,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/12af446c9c7540148d58ff5a40825f0a to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/12af446c9c7540148d58ff5a40825f0a 2023-04-03 02:56:46,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4a53859e3a6941569076eaf7c1537e57 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4a53859e3a6941569076eaf7c1537e57 2023-04-03 02:56:46,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/126d31d5a29641009302f73a8fdb591e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/126d31d5a29641009302f73a8fdb591e 2023-04-03 02:56:46,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/45c708f1e69c4058b3d06c84ae12b7b1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/45c708f1e69c4058b3d06c84ae12b7b1 2023-04-03 02:56:46,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/bbcf346109a249c39594106161cee938 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/bbcf346109a249c39594106161cee938 2023-04-03 02:56:46,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/b0fda944c32b4089a69791f828e84b29 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/b0fda944c32b4089a69791f828e84b29 2023-04-03 02:56:46,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/caa9877a455d40848f361b7eb91ebbda to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/caa9877a455d40848f361b7eb91ebbda 2023-04-03 02:56:46,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/562cc14f7ac149789e8668b102dde4c3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/562cc14f7ac149789e8668b102dde4c3 2023-04-03 02:56:46,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4103a25b6ca54d478dca5d31e1b9f84b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/4103a25b6ca54d478dca5d31e1b9f84b 2023-04-03 02:56:46,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/aa2b426de8804eac91b2f7b4169fa12d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/aa2b426de8804eac91b2f7b4169fa12d 2023-04-03 02:56:46,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/44fb0aae6e014b91a04d8ad46c05fae9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/11c904578db74b468977d3629d8e1891, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f8525d4a53a4dd0b770b668f38a1df6, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/d0c1c43d29274430be664457fc1822b4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ea781834d4764c6a9e778aa5b65a9929, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2bc840572eff4c2c82d930ddeb621514, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/b0da13f0c0eb48c695512326510d0901, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e5e9887304414fea962a7dae11ebbe4d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e206d467bd66418a8e1f61ec81965c76, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f07b41fd7604e7ebe584eafb1c9bd86, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/58ce3d87840949f481b6df724f8db2f4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/7b103572272f47ae8b2212962783e762, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/081d3e5cd10041e1ab69f7a509e2075d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/98470cd74ff64317911cebf4b5349008, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/a25c81f5ec6643989e7521034e303ef9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/97c9b7ea81f5422eb767be7d5b869556, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/6681df7ddeba44b49807af96378cbbee, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/64f6537b11ba479fb2b1ae907fa367a3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/0c9cbb3285f74632b733fcf9311bf0c0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/370dd50ba7e544748e4db8d0baa85365, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/85ebc1ace59149318d6647ceedbafe97, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/57e4a60eb1fd4834b5656e359a2eda8d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ca0cc23cef8541d3ade2ccb72dd730dc, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/220d437b71fe4acd9c0c43f89cd328e3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/591fee175bc6476993fe228b53040e38, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/832782a2a1634a109eca1c6a171101cd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2210235840b94a32a10dc8f600d50cf2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/445b5f8f3c424912bd7657f9c5f33ad0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e41a4708c1fe4bfea56d057fc8da044e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/fc1ecdf7e6df4758a3d6366588edf62e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/906168292d7d4ac69a57f4f6c10e8225, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/a784ac663c4c4de7912758a39dba703e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ffd84227750e4af6b10478e098356996, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/72416bac7cb04390971671ed50912ce1, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/432c15800fa846afb4d559048a4878b8] to archive 2023-04-03 02:56:46,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:56:46,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/44fb0aae6e014b91a04d8ad46c05fae9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/44fb0aae6e014b91a04d8ad46c05fae9 2023-04-03 02:56:46,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/11c904578db74b468977d3629d8e1891 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/11c904578db74b468977d3629d8e1891 2023-04-03 02:56:46,370 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f8525d4a53a4dd0b770b668f38a1df6 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f8525d4a53a4dd0b770b668f38a1df6 2023-04-03 02:56:46,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/d0c1c43d29274430be664457fc1822b4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/d0c1c43d29274430be664457fc1822b4 2023-04-03 02:56:46,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ea781834d4764c6a9e778aa5b65a9929 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ea781834d4764c6a9e778aa5b65a9929 2023-04-03 02:56:46,372 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2bc840572eff4c2c82d930ddeb621514 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2bc840572eff4c2c82d930ddeb621514 2023-04-03 02:56:46,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/b0da13f0c0eb48c695512326510d0901 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/b0da13f0c0eb48c695512326510d0901 2023-04-03 02:56:46,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e5e9887304414fea962a7dae11ebbe4d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e5e9887304414fea962a7dae11ebbe4d 2023-04-03 02:56:46,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e206d467bd66418a8e1f61ec81965c76 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e206d467bd66418a8e1f61ec81965c76 2023-04-03 02:56:46,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f07b41fd7604e7ebe584eafb1c9bd86 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/3f07b41fd7604e7ebe584eafb1c9bd86 2023-04-03 02:56:46,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/58ce3d87840949f481b6df724f8db2f4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/58ce3d87840949f481b6df724f8db2f4 2023-04-03 02:56:46,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/7b103572272f47ae8b2212962783e762 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/7b103572272f47ae8b2212962783e762 2023-04-03 02:56:46,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/081d3e5cd10041e1ab69f7a509e2075d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/081d3e5cd10041e1ab69f7a509e2075d 2023-04-03 02:56:46,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/98470cd74ff64317911cebf4b5349008 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/98470cd74ff64317911cebf4b5349008 2023-04-03 02:56:46,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/a25c81f5ec6643989e7521034e303ef9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/a25c81f5ec6643989e7521034e303ef9 2023-04-03 02:56:46,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/97c9b7ea81f5422eb767be7d5b869556 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/97c9b7ea81f5422eb767be7d5b869556 2023-04-03 02:56:46,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/6681df7ddeba44b49807af96378cbbee to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/6681df7ddeba44b49807af96378cbbee 2023-04-03 02:56:46,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/64f6537b11ba479fb2b1ae907fa367a3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/64f6537b11ba479fb2b1ae907fa367a3 2023-04-03 02:56:46,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/0c9cbb3285f74632b733fcf9311bf0c0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/0c9cbb3285f74632b733fcf9311bf0c0 2023-04-03 02:56:46,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/370dd50ba7e544748e4db8d0baa85365 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/370dd50ba7e544748e4db8d0baa85365 2023-04-03 02:56:46,385 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/85ebc1ace59149318d6647ceedbafe97 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/85ebc1ace59149318d6647ceedbafe97 2023-04-03 02:56:46,386 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/57e4a60eb1fd4834b5656e359a2eda8d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/57e4a60eb1fd4834b5656e359a2eda8d 2023-04-03 02:56:46,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ca0cc23cef8541d3ade2ccb72dd730dc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ca0cc23cef8541d3ade2ccb72dd730dc 2023-04-03 02:56:46,388 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/220d437b71fe4acd9c0c43f89cd328e3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/220d437b71fe4acd9c0c43f89cd328e3 2023-04-03 02:56:46,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/591fee175bc6476993fe228b53040e38 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/591fee175bc6476993fe228b53040e38 2023-04-03 02:56:46,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/832782a2a1634a109eca1c6a171101cd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/832782a2a1634a109eca1c6a171101cd 2023-04-03 02:56:46,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2210235840b94a32a10dc8f600d50cf2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/2210235840b94a32a10dc8f600d50cf2 2023-04-03 02:56:46,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/445b5f8f3c424912bd7657f9c5f33ad0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/445b5f8f3c424912bd7657f9c5f33ad0 2023-04-03 02:56:46,392 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e41a4708c1fe4bfea56d057fc8da044e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/e41a4708c1fe4bfea56d057fc8da044e 2023-04-03 02:56:46,392 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/fc1ecdf7e6df4758a3d6366588edf62e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/fc1ecdf7e6df4758a3d6366588edf62e 2023-04-03 02:56:46,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/906168292d7d4ac69a57f4f6c10e8225 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/906168292d7d4ac69a57f4f6c10e8225 2023-04-03 02:56:46,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/a784ac663c4c4de7912758a39dba703e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/a784ac663c4c4de7912758a39dba703e 2023-04-03 02:56:46,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ffd84227750e4af6b10478e098356996 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/ffd84227750e4af6b10478e098356996 2023-04-03 02:56:46,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/72416bac7cb04390971671ed50912ce1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/72416bac7cb04390971671ed50912ce1 2023-04-03 02:56:46,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/432c15800fa846afb4d559048a4878b8 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/432c15800fa846afb4d559048a4878b8 2023-04-03 02:56:46,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5abb02692aef454894da63309923768f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/57a6578d1ab94c3382e7b084936a7bcb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/467baf9bbf5b41f99885650f2665e720, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/a0e09a82a89948d6b279768066016bd0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/216b9616d0e34e068792beb0bb35e512, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b61a7ce8608f407b962e8e18b2a8a9b9, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/4227b0172b9b4bd28ab844befe445bfb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/648a248e37934103bf18ee14287370ab, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/92437813621447509aa7b129cebef9b3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/1856f681960f43d0811c30b35fa56d0f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2171249d0b314b0aa6dbc99b6959d0ec, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/e260f62d87df4e119066897a7c25673b, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/391160c5d50d4e8e9a4c62eb10b96714, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7355f6ad10914002a3f8e5fa3cd65f91, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/d031b98e4f414dc988b8e59c189a545f, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/143a8567b9fa4363a52f8843e34704c0, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/98028ab3ab524315a965f7f91d2b0061, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c1b41041271e439dbe3caed62b43e5bd, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c53b7b99bf1b4c2f9b0f89552243e9cb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/837836bd13f74c619bdb2201e9abb941, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b5f2987e89da4cd8bd80ab7da5a452bb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/ef50a5f70fff4b7e8b0f84fe61cb4d0e, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c8635e38ed3249ae8ec187dace9f8da3, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7fe5b47aafa244bf83a2abd1d51dd1aa, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/67aae3b8be8841ca9c28a8606c0132ad, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/25148ff61ad94b29855018a57cfcade2, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7e2e38c47a2b4a92bb9e230188eb3a24, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2b6db8826e3e4280adf9be9d2db756b4, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/6daf5f1bb08345c7bd99d2c8ebe7a4cf, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2aa515a4d2e9439a9c651ea82b0a8cfb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b1943e790c84460980cf29d3e2216d6d, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/eaa18c52b13b49078b8e2ca16b7d51cb, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/9a0736034c3546698df3f19dc1e87224, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5edebd48ffb24c19a080052d75933b87, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7778f100083a4a49a9d7dd9798311618] to archive 2023-04-03 02:56:46,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 02:56:46,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5abb02692aef454894da63309923768f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5abb02692aef454894da63309923768f 2023-04-03 02:56:46,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/57a6578d1ab94c3382e7b084936a7bcb to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/57a6578d1ab94c3382e7b084936a7bcb 2023-04-03 02:56:46,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/467baf9bbf5b41f99885650f2665e720 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/467baf9bbf5b41f99885650f2665e720 2023-04-03 02:56:46,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/a0e09a82a89948d6b279768066016bd0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/a0e09a82a89948d6b279768066016bd0 2023-04-03 02:56:46,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/216b9616d0e34e068792beb0bb35e512 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/216b9616d0e34e068792beb0bb35e512 2023-04-03 02:56:46,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b61a7ce8608f407b962e8e18b2a8a9b9 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b61a7ce8608f407b962e8e18b2a8a9b9 2023-04-03 02:56:46,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/4227b0172b9b4bd28ab844befe445bfb to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/4227b0172b9b4bd28ab844befe445bfb 2023-04-03 02:56:46,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/648a248e37934103bf18ee14287370ab to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/648a248e37934103bf18ee14287370ab 2023-04-03 02:56:46,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/92437813621447509aa7b129cebef9b3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/92437813621447509aa7b129cebef9b3 2023-04-03 02:56:46,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/1856f681960f43d0811c30b35fa56d0f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/1856f681960f43d0811c30b35fa56d0f 2023-04-03 02:56:46,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2171249d0b314b0aa6dbc99b6959d0ec to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2171249d0b314b0aa6dbc99b6959d0ec 2023-04-03 02:56:46,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/e260f62d87df4e119066897a7c25673b to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/e260f62d87df4e119066897a7c25673b 2023-04-03 02:56:46,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/391160c5d50d4e8e9a4c62eb10b96714 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/391160c5d50d4e8e9a4c62eb10b96714 2023-04-03 02:56:46,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7355f6ad10914002a3f8e5fa3cd65f91 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7355f6ad10914002a3f8e5fa3cd65f91 2023-04-03 02:56:46,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/d031b98e4f414dc988b8e59c189a545f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/d031b98e4f414dc988b8e59c189a545f 2023-04-03 02:56:46,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/143a8567b9fa4363a52f8843e34704c0 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/143a8567b9fa4363a52f8843e34704c0 2023-04-03 02:56:46,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/98028ab3ab524315a965f7f91d2b0061 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/98028ab3ab524315a965f7f91d2b0061 2023-04-03 02:56:46,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c1b41041271e439dbe3caed62b43e5bd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c1b41041271e439dbe3caed62b43e5bd 2023-04-03 02:56:46,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c53b7b99bf1b4c2f9b0f89552243e9cb to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c53b7b99bf1b4c2f9b0f89552243e9cb 2023-04-03 02:56:46,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/837836bd13f74c619bdb2201e9abb941 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/837836bd13f74c619bdb2201e9abb941 2023-04-03 02:56:46,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b5f2987e89da4cd8bd80ab7da5a452bb to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b5f2987e89da4cd8bd80ab7da5a452bb 2023-04-03 02:56:46,414 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/ef50a5f70fff4b7e8b0f84fe61cb4d0e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/ef50a5f70fff4b7e8b0f84fe61cb4d0e 2023-04-03 02:56:46,414 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c8635e38ed3249ae8ec187dace9f8da3 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/c8635e38ed3249ae8ec187dace9f8da3 2023-04-03 02:56:46,415 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7fe5b47aafa244bf83a2abd1d51dd1aa to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7fe5b47aafa244bf83a2abd1d51dd1aa 2023-04-03 02:56:46,416 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/67aae3b8be8841ca9c28a8606c0132ad to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/67aae3b8be8841ca9c28a8606c0132ad 2023-04-03 02:56:46,416 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/25148ff61ad94b29855018a57cfcade2 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/25148ff61ad94b29855018a57cfcade2 2023-04-03 02:56:46,417 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7e2e38c47a2b4a92bb9e230188eb3a24 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7e2e38c47a2b4a92bb9e230188eb3a24 2023-04-03 02:56:46,418 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2b6db8826e3e4280adf9be9d2db756b4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2b6db8826e3e4280adf9be9d2db756b4 2023-04-03 02:56:46,418 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/6daf5f1bb08345c7bd99d2c8ebe7a4cf to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/6daf5f1bb08345c7bd99d2c8ebe7a4cf 2023-04-03 02:56:46,419 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2aa515a4d2e9439a9c651ea82b0a8cfb to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/2aa515a4d2e9439a9c651ea82b0a8cfb 2023-04-03 02:56:46,420 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b1943e790c84460980cf29d3e2216d6d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/b1943e790c84460980cf29d3e2216d6d 2023-04-03 02:56:46,420 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/eaa18c52b13b49078b8e2ca16b7d51cb to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/eaa18c52b13b49078b8e2ca16b7d51cb 2023-04-03 02:56:46,421 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/9a0736034c3546698df3f19dc1e87224 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/9a0736034c3546698df3f19dc1e87224 2023-04-03 02:56:46,422 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5edebd48ffb24c19a080052d75933b87 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/5edebd48ffb24c19a080052d75933b87 2023-04-03 02:56:46,422 DEBUG [StoreCloser-TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7778f100083a4a49a9d7dd9798311618 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/7778f100083a4a49a9d7dd9798311618 2023-04-03 02:56:46,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/recovered.edits/549.seqid, newMaxSeqId=549, maxSeqId=4 2023-04-03 02:56:46,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b. 2023-04-03 02:56:46,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1af30d70f4e72ebe50e43fde14e4fe9b: 2023-04-03 02:56:46,428 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:46,428 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=1af30d70f4e72ebe50e43fde14e4fe9b, regionState=CLOSED 2023-04-03 02:56:46,428 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680490606428"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680490606428"}]},"ts":"1680490606428"} 2023-04-03 02:56:46,430 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-03 02:56:46,430 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 1af30d70f4e72ebe50e43fde14e4fe9b, server=jenkins-hbase4.apache.org,43815,1680490421348 in 1.0950 sec 2023-04-03 02:56:46,431 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-03 02:56:46,431 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1af30d70f4e72ebe50e43fde14e4fe9b, UNASSIGN in 1.0990 sec 2023-04-03 02:56:46,431 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680490606431"}]},"ts":"1680490606431"} 2023-04-03 02:56:46,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 02:56:46,432 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 02:56:46,433 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 02:56:46,434 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1070 sec 2023-04-03 02:56:47,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 02:56:47,432 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-03 02:56:47,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 02:56:47,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:47,435 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:47,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-03 02:56:47,435 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:47,438 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:47,440 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C, FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/recovered.edits] 2023-04-03 02:56:47,442 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/3941a9f163564843b2d28c01b6a450e4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/3941a9f163564843b2d28c01b6a450e4 2023-04-03 02:56:47,443 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/d27b05a4bcac4ee6a26d32833e6fe25f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/A/d27b05a4bcac4ee6a26d32833e6fe25f 2023-04-03 02:56:47,445 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/524843df737146d18f2cb7d8d08a2b46 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/524843df737146d18f2cb7d8d08a2b46 2023-04-03 02:56:47,446 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/c51a274560bc46568c9d22ce0dbc7705 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/B/c51a274560bc46568c9d22ce0dbc7705 2023-04-03 02:56:47,447 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/0a78215ae6884ae1b87c621461cde67d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/0a78215ae6884ae1b87c621461cde67d 2023-04-03 02:56:47,448 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/e8b03e3b270748b4bfb6541856ed8518 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/C/e8b03e3b270748b4bfb6541856ed8518 2023-04-03 02:56:47,450 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/recovered.edits/549.seqid to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b/recovered.edits/549.seqid 2023-04-03 02:56:47,450 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/.tmp/data/default/TestAcidGuarantees/1af30d70f4e72ebe50e43fde14e4fe9b 2023-04-03 02:56:47,451 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 02:56:47,451 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 02:56:47,451 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-03 02:56:47,454 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403043e767c37ce4476a0e7986c2cc3b15f to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403043e767c37ce4476a0e7986c2cc3b15f 2023-04-03 02:56:47,454 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040306927f55d7fb46648a0d560c1772d5f5 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040306927f55d7fb46648a0d560c1772d5f5 2023-04-03 02:56:47,455 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040313317be73adf4cee85fdbaadbfe26d3d to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040313317be73adf4cee85fdbaadbfe26d3d 2023-04-03 02:56:47,456 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403217b47190142476bb064300e09b37f75 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403217b47190142476bb064300e09b37f75 2023-04-03 02:56:47,457 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403329145d337934fee974795953efd3cb7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403329145d337934fee974795953efd3cb7 2023-04-03 02:56:47,458 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403348447b4e9eb48c2ab761ea4370ce244 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403348447b4e9eb48c2ab761ea4370ce244 2023-04-03 02:56:47,458 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403369a4d6f9433405f8c65302362e16efa to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403369a4d6f9433405f8c65302362e16efa 2023-04-03 02:56:47,459 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040336b10d80ae504ab2b9b4bd60021f2b1e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040336b10d80ae504ab2b9b4bd60021f2b1e 2023-04-03 02:56:47,460 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033a515758d48e43adb84653b333ef6db4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033a515758d48e43adb84653b333ef6db4 2023-04-03 02:56:47,461 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033bb63f7cd62b46e1a761d1a990ade5fd to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033bb63f7cd62b46e1a761d1a990ade5fd 2023-04-03 02:56:47,462 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403495cb8ced0724e5b86fd15ec1b154eb4 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403495cb8ced0724e5b86fd15ec1b154eb4 2023-04-03 02:56:47,463 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034d584f75555248e19fabfc339d7e8f94 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034d584f75555248e19fabfc339d7e8f94 2023-04-03 02:56:47,463 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036a26bf0d05924abc8e5dc4748ff748ce to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036a26bf0d05924abc8e5dc4748ff748ce 2023-04-03 02:56:47,464 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040374ae96c252ae457dbf018cd2b5d89933 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040374ae96c252ae457dbf018cd2b5d89933 2023-04-03 02:56:47,465 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040382331f45cefd4696ad73231c9f23f2bc to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040382331f45cefd4696ad73231c9f23f2bc 2023-04-03 02:56:47,466 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038c97a356dea441269eacbb7241b1722c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038c97a356dea441269eacbb7241b1722c 2023-04-03 02:56:47,467 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038d0248ca17ce429eb8db8c34eb86eeed to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038d0248ca17ce429eb8db8c34eb86eeed 2023-04-03 02:56:47,468 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040399e3ffb8c415468c810a105edf90c849 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040399e3ffb8c415468c810a105edf90c849 2023-04-03 02:56:47,469 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039d8e0cd2bcb64ab9a63daa674bce897c to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039d8e0cd2bcb64ab9a63daa674bce897c 2023-04-03 02:56:47,469 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a0a8fa6a80534d988c0184e92ee77651 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a0a8fa6a80534d988c0184e92ee77651 2023-04-03 02:56:47,470 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b8fa3595469f42d4b92b31051644ebd7 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b8fa3595469f42d4b92b31051644ebd7 2023-04-03 02:56:47,471 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bc17fca84a734229b6f3bc74535be2f1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bc17fca84a734229b6f3bc74535be2f1 2023-04-03 02:56:47,472 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bee0e2eb5249444fba0005d5a59237b1 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bee0e2eb5249444fba0005d5a59237b1 2023-04-03 02:56:47,473 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c1ce54c5e0cb4b27bef8a2eba4800007 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c1ce54c5e0cb4b27bef8a2eba4800007 2023-04-03 02:56:47,474 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e10d32a58c114019a31e51300202d9fe to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e10d32a58c114019a31e51300202d9fe 2023-04-03 02:56:47,474 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ee8bc690afc64e20bdd8f2cf4616ae80 to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ee8bc690afc64e20bdd8f2cf4616ae80 2023-04-03 02:56:47,475 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f1d335738d6b4e2aaca484c9dc96d25e to hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f1d335738d6b4e2aaca484c9dc96d25e 2023-04-03 02:56:47,476 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 02:56:47,477 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:47,478 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 02:56:47,480 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 02:56:47,480 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:47,480 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 02:56:47,481 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680490607481"}]},"ts":"9223372036854775807"} 2023-04-03 02:56:47,482 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 02:56:47,482 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1af30d70f4e72ebe50e43fde14e4fe9b, NAME => 'TestAcidGuarantees,,1680490578416.1af30d70f4e72ebe50e43fde14e4fe9b.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 02:56:47,482 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 02:56:47,482 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680490607482"}]},"ts":"9223372036854775807"} 2023-04-03 02:56:47,483 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 02:56:47,484 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 02:56:47,485 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 51 msec 2023-04-03 02:56:47,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37975] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-03 02:56:47,536 INFO [Listener at localhost/46853] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-03 02:56:47,549 INFO [Listener at localhost/46853] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227 (was 231), OpenFileDescriptor=578 (was 563) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=436 (was 504), ProcessCount=170 (was 170), AvailableMemoryMB=3726 (was 3907) 2023-04-03 02:56:47,549 INFO [Listener at localhost/46853] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-03 02:56:47,550 INFO [Listener at localhost/46853] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 02:56:47,550 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f3002be to 127.0.0.1:52506 2023-04-03 02:56:47,550 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:47,550 DEBUG [Listener at localhost/46853] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-03 02:56:47,550 DEBUG [Listener at localhost/46853] util.JVMClusterUtil(257): Found active master hash=279410515, stopped=false 2023-04-03 02:56:47,550 INFO [Listener at localhost/46853] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,37975,1680490420046 2023-04-03 02:56:47,552 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-03 02:56:47,552 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-03 02:56:47,552 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 02:56:47,552 INFO [Listener at localhost/46853] procedure2.ProcedureExecutor(629): Stopping 2023-04-03 02:56:47,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 02:56:47,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 02:56:47,553 DEBUG [Listener at localhost/46853] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f2e809b to 127.0.0.1:52506 2023-04-03 02:56:47,553 DEBUG [Listener at localhost/46853] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:47,553 INFO [Listener at localhost/46853] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43815,1680490421348' ***** 2023-04-03 02:56:47,553 INFO [Listener at localhost/46853] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-03 02:56:47,554 INFO [RS:0;jenkins-hbase4:43815] regionserver.HeapMemoryManager(220): Stopping 2023-04-03 02:56:47,554 INFO [RS:0;jenkins-hbase4:43815] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-03 02:56:47,554 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-03 02:56:47,554 INFO [RS:0;jenkins-hbase4:43815] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-03 02:56:47,554 INFO [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(3304): Received CLOSE for 624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:56:47,554 INFO [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:47,554 DEBUG [RS:0;jenkins-hbase4:43815] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7129712f to 127.0.0.1:52506 2023-04-03 02:56:47,554 DEBUG [RS:0;jenkins-hbase4:43815] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:47,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 624516272efe3dcb8cad0effd67e2ce3, disabling compactions & flushes 2023-04-03 02:56:47,555 INFO [RS:0;jenkins-hbase4:43815] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-03 02:56:47,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:56:47,555 INFO [RS:0;jenkins-hbase4:43815] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-03 02:56:47,555 INFO [RS:0;jenkins-hbase4:43815] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-03 02:56:47,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:56:47,555 INFO [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-03 02:56:47,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. after waiting 0 ms 2023-04-03 02:56:47,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:56:47,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 624516272efe3dcb8cad0effd67e2ce3 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-03 02:56:47,555 INFO [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-03 02:56:47,556 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 624516272efe3dcb8cad0effd67e2ce3=hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3.} 2023-04-03 02:56:47,556 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-03 02:56:47,556 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-03 02:56:47,556 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-03 02:56:47,556 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-03 02:56:47,556 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-03 02:56:47,556 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-03 02:56:47,558 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1505): Waiting on 1588230740, 624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:56:47,591 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-03 02:56:47,647 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-03 02:56:47,647 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-03 02:56:47,758 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1505): Waiting on 1588230740, 624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:56:47,958 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1505): Waiting on 1588230740, 624516272efe3dcb8cad0effd67e2ce3 2023-04-03 02:56:47,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3/.tmp/info/881c52036f384939a8a8a950ce147cb1 2023-04-03 02:56:47,965 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/.tmp/info/f971b1fc30494207a72dfa4a806a5a5e 2023-04-03 02:56:47,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3/.tmp/info/881c52036f384939a8a8a950ce147cb1 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3/info/881c52036f384939a8a8a950ce147cb1 2023-04-03 02:56:47,970 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f971b1fc30494207a72dfa4a806a5a5e 2023-04-03 02:56:47,970 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3/info/881c52036f384939a8a8a950ce147cb1, entries=2, sequenceid=6, filesize=4.8 K 2023-04-03 02:56:47,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 624516272efe3dcb8cad0effd67e2ce3 in 416ms, sequenceid=6, compaction requested=false 2023-04-03 02:56:47,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/namespace/624516272efe3dcb8cad0effd67e2ce3/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-03 02:56:47,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:56:47,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 624516272efe3dcb8cad0effd67e2ce3: 2023-04-03 02:56:47,983 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680490423577.624516272efe3dcb8cad0effd67e2ce3. 2023-04-03 02:56:48,158 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 02:56:48,359 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 02:56:48,383 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/.tmp/rep_barrier/88195909a01a401ab1fde37ce066b5c8 2023-04-03 02:56:48,386 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 88195909a01a401ab1fde37ce066b5c8 2023-04-03 02:56:48,559 INFO [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-03 02:56:48,559 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-03 02:56:48,559 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 02:56:48,759 DEBUG [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 02:56:48,792 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/.tmp/table/f20b961497f64ceda4fd5fa424ff6785 2023-04-03 02:56:48,795 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f20b961497f64ceda4fd5fa424ff6785 2023-04-03 02:56:48,796 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/.tmp/info/f971b1fc30494207a72dfa4a806a5a5e as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/info/f971b1fc30494207a72dfa4a806a5a5e 2023-04-03 02:56:48,799 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f971b1fc30494207a72dfa4a806a5a5e 2023-04-03 02:56:48,799 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/info/f971b1fc30494207a72dfa4a806a5a5e, entries=22, sequenceid=93, filesize=7.1 K 2023-04-03 02:56:48,801 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/.tmp/rep_barrier/88195909a01a401ab1fde37ce066b5c8 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/rep_barrier/88195909a01a401ab1fde37ce066b5c8 2023-04-03 02:56:48,804 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 88195909a01a401ab1fde37ce066b5c8 2023-04-03 02:56:48,805 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/rep_barrier/88195909a01a401ab1fde37ce066b5c8, entries=6, sequenceid=93, filesize=5.5 K 2023-04-03 02:56:48,805 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/.tmp/table/f20b961497f64ceda4fd5fa424ff6785 as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/table/f20b961497f64ceda4fd5fa424ff6785 2023-04-03 02:56:48,809 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f20b961497f64ceda4fd5fa424ff6785 2023-04-03 02:56:48,809 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/table/f20b961497f64ceda4fd5fa424ff6785, entries=9, sequenceid=93, filesize=5.5 K 2023-04-03 02:56:48,810 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1254ms, sequenceid=93, compaction requested=false 2023-04-03 02:56:48,816 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-03 02:56:48,817 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-03 02:56:48,818 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-03 02:56:48,818 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-03 02:56:48,818 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-03 02:56:48,959 INFO [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43815,1680490421348; all regions closed. 2023-04-03 02:56:48,967 DEBUG [RS:0;jenkins-hbase4:43815] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/oldWALs 2023-04-03 02:56:48,968 INFO [RS:0;jenkins-hbase4:43815] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43815%2C1680490421348.meta:.meta(num 1680490423328) 2023-04-03 02:56:48,971 DEBUG [RS:0;jenkins-hbase4:43815] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/oldWALs 2023-04-03 02:56:48,972 INFO [RS:0;jenkins-hbase4:43815] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43815%2C1680490421348:(num 1680490422756) 2023-04-03 02:56:48,972 DEBUG [RS:0;jenkins-hbase4:43815] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:48,972 INFO [RS:0;jenkins-hbase4:43815] regionserver.LeaseManager(133): Closed leases 2023-04-03 02:56:48,972 INFO [RS:0;jenkins-hbase4:43815] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-03 02:56:48,972 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-03 02:56:48,974 INFO [RS:0;jenkins-hbase4:43815] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43815 2023-04-03 02:56:48,980 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43815,1680490421348 2023-04-03 02:56:48,980 ERROR [Listener at localhost/46853-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@87cf5 rejected from java.util.concurrent.ThreadPoolExecutor@437e82c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2322] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 02:56:48,980 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-03 02:56:48,981 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-03 02:56:48,981 ERROR [Listener at localhost/46853-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3392d8d7 rejected from java.util.concurrent.ThreadPoolExecutor@437e82c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2322] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 02:56:48,984 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43815,1680490421348] 2023-04-03 02:56:48,985 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43815,1680490421348; numProcessing=1 2023-04-03 02:56:48,986 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43815,1680490421348 already deleted, retry=false 2023-04-03 02:56:48,986 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,43815,1680490421348 expired; onlineServers=0 2023-04-03 02:56:48,986 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37975,1680490420046' ***** 2023-04-03 02:56:48,986 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-03 02:56:48,986 DEBUG [M:0;jenkins-hbase4:37975] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@519a7d68, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-03 02:56:48,986 INFO [M:0;jenkins-hbase4:37975] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37975,1680490420046 2023-04-03 02:56:48,986 INFO [M:0;jenkins-hbase4:37975] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37975,1680490420046; all regions closed. 2023-04-03 02:56:48,986 DEBUG [M:0;jenkins-hbase4:37975] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 02:56:48,986 DEBUG [M:0;jenkins-hbase4:37975] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-03 02:56:48,986 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-03 02:56:48,986 DEBUG [M:0;jenkins-hbase4:37975] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-03 02:56:48,986 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680490422460] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680490422460,5,FailOnTimeoutGroup] 2023-04-03 02:56:48,986 INFO [M:0;jenkins-hbase4:37975] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-03 02:56:48,986 INFO [M:0;jenkins-hbase4:37975] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-03 02:56:48,986 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680490422460] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680490422460,5,FailOnTimeoutGroup] 2023-04-03 02:56:48,987 INFO [M:0;jenkins-hbase4:37975] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-03 02:56:48,987 DEBUG [M:0;jenkins-hbase4:37975] master.HMaster(1512): Stopping service threads 2023-04-03 02:56:48,987 INFO [M:0;jenkins-hbase4:37975] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-03 02:56:48,987 ERROR [M:0;jenkins-hbase4:37975] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-03 02:56:48,988 INFO [M:0;jenkins-hbase4:37975] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-03 02:56:48,988 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-03 02:56:48,988 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-03 02:56:48,988 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 02:56:48,989 DEBUG [M:0;jenkins-hbase4:37975] zookeeper.ZKUtil(398): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-03 02:56:48,989 WARN [M:0;jenkins-hbase4:37975] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-03 02:56:48,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-03 02:56:48,989 INFO [M:0;jenkins-hbase4:37975] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-03 02:56:48,989 INFO [M:0;jenkins-hbase4:37975] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-03 02:56:48,989 DEBUG [M:0;jenkins-hbase4:37975] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-03 02:56:48,989 INFO [M:0;jenkins-hbase4:37975] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 02:56:48,989 DEBUG [M:0;jenkins-hbase4:37975] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 02:56:48,989 DEBUG [M:0;jenkins-hbase4:37975] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-03 02:56:48,989 DEBUG [M:0;jenkins-hbase4:37975] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 02:56:48,990 INFO [M:0;jenkins-hbase4:37975] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.83 KB heapSize=393.54 KB 2023-04-03 02:56:49,001 INFO [M:0;jenkins-hbase4:37975] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.83 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/3b432295e02e490ca568058903abe13c 2023-04-03 02:56:49,004 INFO [M:0;jenkins-hbase4:37975] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3b432295e02e490ca568058903abe13c 2023-04-03 02:56:49,005 DEBUG [M:0;jenkins-hbase4:37975] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/3b432295e02e490ca568058903abe13c as hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/3b432295e02e490ca568058903abe13c 2023-04-03 02:56:49,008 INFO [M:0;jenkins-hbase4:37975] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3b432295e02e490ca568058903abe13c 2023-04-03 02:56:49,009 INFO [M:0;jenkins-hbase4:37975] regionserver.HStore(1080): Added hdfs://localhost:37791/user/jenkins/test-data/7bda594f-ce37-14c9-a76b-66198cf3116c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/3b432295e02e490ca568058903abe13c, entries=65, sequenceid=578, filesize=24.7 K 2023-04-03 02:56:49,009 INFO [M:0;jenkins-hbase4:37975] regionserver.HRegion(2948): Finished flush of dataSize ~344.83 KB/353106, heapSize ~393.52 KB/402968, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 20ms, sequenceid=578, compaction requested=false 2023-04-03 02:56:49,010 INFO [M:0;jenkins-hbase4:37975] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 02:56:49,010 DEBUG [M:0;jenkins-hbase4:37975] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-03 02:56:49,013 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-03 02:56:49,013 INFO [M:0;jenkins-hbase4:37975] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-03 02:56:49,014 INFO [M:0;jenkins-hbase4:37975] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37975 2023-04-03 02:56:49,015 DEBUG [M:0;jenkins-hbase4:37975] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,37975,1680490420046 already deleted, retry=false 2023-04-03 02:56:49,082 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 02:56:49,082 INFO [RS:0;jenkins-hbase4:43815] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43815,1680490421348; zookeeper connection closed. 2023-04-03 02:56:49,083 ERROR [Listener at localhost/46853-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4bf30537 rejected from java.util.concurrent.ThreadPoolExecutor@437e82c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2322] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 02:56:49,083 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): regionserver:43815-0x10147f0bdc10001, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 02:56:49,083 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@438ccff6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@438ccff6 2023-04-03 02:56:49,083 ERROR [Listener at localhost/46853-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@46bb915 rejected from java.util.concurrent.ThreadPoolExecutor@437e82c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2322] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 02:56:49,084 INFO [Listener at localhost/46853] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-03 02:56:49,182 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 02:56:49,182 INFO [M:0;jenkins-hbase4:37975] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37975,1680490420046; zookeeper connection closed. 2023-04-03 02:56:49,182 ERROR [Listener at localhost/46853-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3353c06a rejected from java.util.concurrent.ThreadPoolExecutor@3d42a1b1[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3885] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 02:56:49,182 DEBUG [Listener at localhost/46853-EventThread] zookeeper.ZKWatcher(600): master:37975-0x10147f0bdc10000, quorum=127.0.0.1:52506, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 02:56:49,182 ERROR [Listener at localhost/46853-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@23dc5dc1 rejected from java.util.concurrent.ThreadPoolExecutor@3d42a1b1[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3885] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 02:56:49,184 WARN [Listener at localhost/46853] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-03 02:56:49,186 INFO [Listener at localhost/46853] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-03 02:56:49,294 WARN [BP-966636340-172.31.14.131-1680490416035 heartbeating to localhost/127.0.0.1:37791] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-03 02:56:49,294 WARN [BP-966636340-172.31.14.131-1680490416035 heartbeating to localhost/127.0.0.1:37791] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-966636340-172.31.14.131-1680490416035 (Datanode Uuid 328bbd6e-f653-4143-b18d-0bb4dc77cf58) service to localhost/127.0.0.1:37791 2023-04-03 02:56:49,297 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/cluster_db440580-0945-d2b2-2bc7-6fe721462712/dfs/data/data1/current/BP-966636340-172.31.14.131-1680490416035] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-03 02:56:49,297 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/641f50bb-bc79-f359-1f93-149811a73022/cluster_db440580-0945-d2b2-2bc7-6fe721462712/dfs/data/data2/current/BP-966636340-172.31.14.131-1680490416035] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-03 02:56:49,331 INFO [Listener at localhost/46853] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-03 02:56:49,504 INFO [Listener at localhost/46853] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-03 02:56:49,682 INFO [Listener at localhost/46853] hbase.HBaseTestingUtility(1293): Minicluster is down